Nov 02 19:47:22.481136 ip-10-0-128-34 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:47:22.943433 ip-10-0-128-34 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:22.943433 ip-10-0-128-34 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:47:22.943433 ip-10-0-128-34 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:22.943433 ip-10-0-128-34 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:22.948999 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.948913 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.948999 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.948999 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.948999 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949003 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949007 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949010 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949012 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949015 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949018 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949021 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949023 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949026 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949029 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949031 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949046 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949049 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949052 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949054 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949057 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949060 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949063 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949065 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.949094 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949068 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949071 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949074 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949076 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949079 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949082 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949085 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949087 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949090 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949093 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949095 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949098 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949101 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949104 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949107 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949110 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949114 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949119 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949123 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.949537 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949126 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949128 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949131 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949134 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949136 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949139 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949141 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949144 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949146 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949149 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949152 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949154 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949156 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949159 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949162 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949165 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949168 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949171 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949174 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949177 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.949992 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949180 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949183 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949186 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949189 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949191 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949194 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949197 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949199 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949202 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949204 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949207 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949210 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949213 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949215 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949218 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949223 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949225 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949228 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949230 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.950457 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949234 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949236 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949239 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949241 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949244 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949247 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949249 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949253 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949256 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949259 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949261 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949264 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949266 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949269 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949271 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949274 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949277 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949279 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949284 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949288 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.950957 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949291 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949294 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949297 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949300 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949302 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.949306 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950964 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950972 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950976 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950978 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950982 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950985 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950988 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950991 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950994 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.950997 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951000 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951003 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951006 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951010 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.951437 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951012 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951015 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951017 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951020 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951022 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951025 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951027 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951030 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951032 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951035 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951037 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951040 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951042 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951045 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951048 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951051 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951054 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951057 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951060 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951063 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.951960 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951066 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951068 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951071 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951073 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951076 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951078 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951081 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951083 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951086 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951089 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951091 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951094 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951097 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951099 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951102 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951105 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951107 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951110 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951113 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.952433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951115 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951118 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951120 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951123 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951126 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951131 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951135 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951138 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951141 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951144 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951147 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951150 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951153 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951156 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951158 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951161 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951164 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951166 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951169 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951172 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.952936 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951174 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951177 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951179 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951182 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951184 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951187 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951190 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951192 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951195 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951198 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951203 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951207 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951216 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951224 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951227 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951229 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951232 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951235 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.953412 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951238 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951241 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951243 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951246 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951250 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951252 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951256 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951258 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951261 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951264 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951266 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951269 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951271 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951274 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951780 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951785 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951788 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951791 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951795 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.953879 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951797 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951802 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951805 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951809 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951811 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951814 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951817 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951819 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951823 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951827 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951830 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951833 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951837 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951840 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951842 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951845 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951848 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951851 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.954338 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951866 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951869 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951872 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951874 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951877 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951879 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951882 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951885 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951887 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951891 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951895 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951897 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951900 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951902 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951905 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951907 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951910 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951913 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951916 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.954888 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951919 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951922 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951925 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951927 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951930 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951932 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951935 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951938 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951941 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951943 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951946 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951949 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951951 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951954 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951957 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951960 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951962 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951965 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951968 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951971 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.955347 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951974 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951977 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951979 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951982 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951984 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951987 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951989 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951992 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951994 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.951997 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952000 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952003 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952005 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952008 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952010 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952013 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952016 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952018 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952021 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.955807 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952024 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952027 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952029 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952031 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952034 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952037 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952039 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952042 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952048 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952051 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952054 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952056 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952059 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952061 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952064 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952066 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952069 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952071 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952074 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.956267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952076 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952080 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952082 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952085 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.952088 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954039 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954055 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954062 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954067 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954072 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954075 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954080 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954084 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954088 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954091 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954095 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954098 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954101 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954105 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954108 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954111 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954114 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954124 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954131 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:47:22.956711 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954134 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954137 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954141 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954144 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954148 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954151 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954155 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954158 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954161 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954165 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954168 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954171 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954175 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954180 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954183 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954186 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954190 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954193 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954195 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954202 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954206 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954209 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954212 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954215 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:47:22.957319 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954219 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954223 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954226 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954229 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954232 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954235 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954238 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954247 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954250 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954253 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954256 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954260 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954263 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954266 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954269 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954272 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954276 2581 flags.go:64] FLAG: --help="false" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954279 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954282 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954286 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954289 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954293 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954296 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:47:22.957939 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954299 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954302 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954305 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954308 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954311 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954315 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954318 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954321 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954324 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954327 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954330 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954333 2581 flags.go:64] FLAG: --lock-file="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954336 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954339 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954343 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954353 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954356 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954365 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954368 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954371 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954375 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954378 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954380 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954385 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954389 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:47:22.958490 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954393 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954396 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954399 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954401 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954405 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954408 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954411 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954414 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954421 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954424 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954427 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954430 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954433 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954436 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954439 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954442 2581 flags.go:64] FLAG: --port="10250" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954445 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954448 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d1f8f1d19832ac46" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954451 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954454 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954457 2581 flags.go:64] FLAG: --register-node="true" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954460 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954464 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954467 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954471 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954475 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:47:22.959091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954479 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954482 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954485 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954488 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954491 2581 flags.go:64] FLAG: --runonce="false" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954494 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954497 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954500 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954503 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954506 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954509 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954512 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954515 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954518 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954521 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954524 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954527 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954531 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954534 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954539 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954545 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954548 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954551 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954556 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954559 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:47:22.959696 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954562 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954565 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954568 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954571 2581 flags.go:64] FLAG: --v="2" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954576 2581 flags.go:64] FLAG: --version="false" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954580 2581 flags.go:64] FLAG: --vmodule="" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954585 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.954589 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954749 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954753 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954757 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954760 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954762 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954765 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954768 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954771 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954774 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954776 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954779 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954783 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954786 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954788 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.960297 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954791 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954794 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954796 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954799 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954803 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954806 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954809 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954811 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954815 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954820 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954823 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954826 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954829 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954831 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954834 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954837 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954840 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954842 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.960944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954850 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954864 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954867 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954870 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954872 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954875 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954878 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954880 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954883 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954886 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954888 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954891 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954894 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954897 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954900 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954903 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954905 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954910 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954913 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.961382 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954916 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954918 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954921 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954924 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954926 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954929 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954931 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954934 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954936 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954939 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954942 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954944 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954948 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954951 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954954 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954957 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954959 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954962 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954965 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.961908 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954967 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954970 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954973 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954975 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954978 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954980 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954983 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954986 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954988 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954991 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954993 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.954997 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955000 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955003 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955005 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955008 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955011 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955013 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955016 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955018 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.962365 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955021 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955023 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955026 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955029 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955031 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955035 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955037 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955041 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955045 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955049 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955051 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955054 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955057 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955059 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955062 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.955090 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:22.962846 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955131 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955135 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955138 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955141 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955145 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955147 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955150 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955154 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955157 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955160 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955162 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955165 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955167 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955172 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955175 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955179 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955183 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955186 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955189 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.963243 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955191 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955194 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955197 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955201 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955203 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955207 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955219 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955222 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955225 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955228 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955231 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955233 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955236 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955239 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955241 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955244 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955247 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955249 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955252 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955255 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.963685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955259 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955261 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955264 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955267 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955270 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955273 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955275 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955278 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955281 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955283 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955286 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955289 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955291 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955294 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955297 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955301 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955303 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955306 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955309 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.964171 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955312 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955315 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955317 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955320 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955323 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955325 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955328 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955330 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955333 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955336 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955338 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955341 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955344 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955348 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955351 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955353 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955356 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955359 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955362 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955364 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.964623 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955367 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955370 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955372 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955375 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955377 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955380 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955383 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955387 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955391 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955393 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955396 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955399 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955403 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955407 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955410 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955413 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955416 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955418 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955421 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955424 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.965172 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955426 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955429 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955432 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955434 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955437 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955441 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955444 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.955447 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.963919 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.963936 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.963952 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.963956 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964015 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964020 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964024 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964028 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964031 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964034 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964037 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964040 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.965624 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964043 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964046 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964049 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964051 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964054 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964056 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964059 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964062 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964064 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964067 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964069 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964072 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964074 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964077 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964079 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964082 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964084 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964087 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964090 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.966109 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964093 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964095 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964099 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964102 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964105 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964108 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964110 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964113 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964115 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964118 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964120 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964123 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964126 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964128 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964131 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964134 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964136 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964140 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964144 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.966598 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964148 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964151 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964154 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964160 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964163 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964166 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964168 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964171 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964174 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964176 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964179 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964182 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964185 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964188 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964191 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964194 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964197 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964200 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964202 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.967130 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964205 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964207 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964210 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964212 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964215 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964217 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964220 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964223 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964225 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964229 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964233 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964236 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964239 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964242 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964245 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964248 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964250 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964253 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964255 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964258 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.967685 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964261 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964264 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964266 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964269 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964272 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964274 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964277 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964279 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964282 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964285 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964287 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964290 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964292 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964295 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964298 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964300 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964303 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964305 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964308 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.968194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964310 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.964338 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964523 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964529 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964532 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964535 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964538 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964541 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964543 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964546 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964548 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964551 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964554 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964558 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964563 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.968742 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964567 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964570 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964572 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964575 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964578 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964580 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964583 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964585 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964594 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964597 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964599 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964602 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964605 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964608 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964610 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964613 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964616 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964623 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964626 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964629 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.969313 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964632 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964634 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964637 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964640 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964643 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964646 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964649 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964651 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964654 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964656 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964659 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964662 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964665 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964667 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964670 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964673 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964675 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964678 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964680 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.970030 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964683 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964685 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964693 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964695 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964698 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964700 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964703 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964705 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964708 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964711 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964713 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964716 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964718 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964721 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964723 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964726 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964731 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964733 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964736 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.970606 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964739 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964741 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964744 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964747 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964749 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964753 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964757 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964759 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964762 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964765 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964767 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964770 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964773 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964775 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964778 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964786 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964789 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964792 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964794 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.971106 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964797 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964799 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964802 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964804 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964807 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964809 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964812 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964814 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964817 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964820 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964822 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964825 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964827 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964830 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964832 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.964875 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:22.971603 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964914 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964918 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964921 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964927 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964930 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964932 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964935 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964937 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964940 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964943 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964945 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964948 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964950 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964957 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964960 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964963 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964965 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964970 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964972 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:22.972003 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964975 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964978 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964980 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964983 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964986 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964988 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964991 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964994 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964996 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.964999 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965001 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965004 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965006 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965009 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965011 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965014 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965017 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965019 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965021 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965024 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:22.972433 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965026 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965029 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965032 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965034 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965036 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965039 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965042 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965049 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965052 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965055 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965060 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965064 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965068 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965070 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965074 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965077 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965079 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965083 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965086 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965088 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:22.972976 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965091 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965094 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965096 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965099 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965102 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965106 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965108 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965111 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965113 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965116 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965118 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965121 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965123 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965126 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965128 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965131 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965133 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965136 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965138 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:22.973487 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965141 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965148 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965151 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965156 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965161 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965163 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965166 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965168 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965171 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965174 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965177 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965181 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965184 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965188 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965191 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965195 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965198 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965202 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:22.973944 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965206 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965208 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965211 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965214 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965216 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965219 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965222 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965226 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:22.965229 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.965291 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.970316 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.971484 2581 server.go:1014] "Starting client certificate rotation" Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.971579 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:47:22.974622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.971609 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:47:22.975194 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.974720 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:47:22.976103 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.976086 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:47:22.988420 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.988405 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:47:22.992048 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.992033 2581 log.go:25] "Validated CRI v1 image API" Nov 02 19:47:22.992791 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.992763 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:47:22.996307 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.996283 2581 fs.go:135] Filesystem UUIDs: map[5114f8f1-5af3-4b4e-8f44-815d5bba4884:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 861710a1-f2b9-4d0d-96f3-faa49fb40a95:/dev/nvme0n1p4] Nov 02 19:47:22.996376 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:22.996307 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:47:23.001632 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.001516 2581 manager.go:217] Machine: {Timestamp:2026-11-02 19:47:22.99997554 +0000 UTC m=+0.387218181 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099782 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b243f0c4127f0be85a12534bfacd SystemUUID:ec23b243-f0c4-127f-0be8-5a12534bfacd BootID:ff49ec97-3a5c-4eb9-b5e2-59d752eb52c0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f2:1b:79:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f2:1b:79:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:cb:99:ea:f4:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:47:23.001632 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.001627 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:47:23.001769 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.001742 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:47:23.005227 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005200 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:47:23.005361 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005228 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-34.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:47:23.005407 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005364 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:47:23.005407 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005371 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:47:23.005407 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005396 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:47:23.005484 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.005408 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:47:23.006376 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.006366 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:47:23.009677 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.009664 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:47:23.009795 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.009786 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:47:23.010769 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.010760 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:47:23.010807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.010772 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:47:23.010807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.010783 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:47:23.010807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.010793 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:47:23.010913 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.010808 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:47:23.011224 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.011202 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:47:23.013813 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.013793 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:47:23.020848 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.020828 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:47:23.021798 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.021785 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:47:23.021865 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.021816 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:47:23.021983 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.021973 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:47:23.022020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.021988 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:47:23.022020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.021994 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:47:23.022020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022000 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:47:23.022020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022006 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:47:23.022020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022011 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:47:23.022723 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022715 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:47:23.022723 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022723 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:47:23.022778 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022731 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:47:23.022778 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022736 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:47:23.022778 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022745 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:47:23.022778 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.022753 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:47:23.024071 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.024061 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:47:23.024071 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.024072 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:47:23.027846 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.027833 2581 server.go:1263] "Started kubelet" Nov 02 19:47:23.027992 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.027922 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:47:23.028154 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.027954 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:47:23.028229 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.028174 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:47:23.028717 ip-10-0-128-34 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:47:23.029650 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.029499 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:47:23.030359 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.030229 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:47:23.034622 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.034600 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:47:23.036323 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.036307 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:47:23.037505 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.037490 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:47:23.037659 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.037637 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:47:23.038307 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038288 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:47:23.038634 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038614 2581 factory.go:153] Registering CRI-O factory Nov 02 19:47:23.038712 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038657 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 19:47:23.038746 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038724 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:47:23.038746 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038734 2581 factory.go:55] Registering systemd factory Nov 02 19:47:23.038746 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038742 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:47:23.038840 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038759 2581 factory.go:103] Registering Raw factory Nov 02 19:47:23.038840 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038775 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 19:47:23.038916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038880 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:47:23.038916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.038898 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:47:23.039028 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.039016 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:47:23.039062 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.039028 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:47:23.039878 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.039840 2581 manager.go:319] Starting recovery of all containers Nov 02 19:47:23.049668 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.049643 2581 manager.go:324] Recovery completed Nov 02 19:47:23.056577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.055485 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.058122 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058102 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.058202 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058134 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.058202 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058145 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.058628 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058613 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:47:23.058678 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058628 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:47:23.058678 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058638 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.058678 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.058648 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:47:23.059318 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.059304 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.059391 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.059331 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.059391 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.059343 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.060227 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.060214 2581 policy_none.go:50] "Start" Nov 02 19:47:23.060264 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.060230 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:47:23.060264 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.060240 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:47:23.061926 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.061915 2581 policy_none.go:44] "Start" Nov 02 19:47:23.109076 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.109048 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:47:23.120278 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.120262 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 19:47:23.121079 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.121052 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:47:23.121079 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121079 2581 server.go:87] "Starting device plugin registration server" Nov 02 19:47:23.121338 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121325 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:47:23.121416 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121339 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:47:23.121504 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121437 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:47:23.121596 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121580 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:47:23.121596 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.121591 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:47:23.122077 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.122060 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:47:23.122175 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.122100 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:47:23.169258 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.162297 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-34.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:47:23.169258 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.162299 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-34.us-west-2.compute.internal.18e4521e9c766e6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-34.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-34.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:23.027803757 +0000 UTC m=+0.415046399,LastTimestamp:2026-11-02 19:47:23.027803757 +0000 UTC m=+0.415046399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-34.us-west-2.compute.internal,}" Nov 02 19:47:23.202394 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.202312 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:47:23.203669 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.203646 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:47:23.203669 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.203669 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:47:23.203830 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.203709 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:47:23.203830 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.203748 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:47:23.208631 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.208603 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-34.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:47:23.221432 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.221414 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.222598 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.222584 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.222662 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.222612 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.222662 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.222626 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.222662 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.222649 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.245932 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.245912 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:47:23.246028 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.245962 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8w49g" Nov 02 19:47:23.304687 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.304657 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal"] Nov 02 19:47:23.304790 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.304730 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.305623 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.305607 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.305723 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.305652 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.305723 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.305666 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.307999 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.307986 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.308154 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.308194 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308170 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.308728 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308713 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.308728 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308724 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.308821 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308741 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.308821 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308747 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.308821 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308755 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.308821 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.308759 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.310021 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.310004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.310116 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.310032 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:23.310767 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.310751 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:23.310840 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.310771 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:23.310840 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.310780 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:23.311404 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.311390 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:23.313220 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.313206 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:23.340026 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.340004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.340125 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.340030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.340125 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.340048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.351779 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.351759 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.354283 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.354266 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.404568 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.404546 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:23.405447 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.405433 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:23.406752 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.406734 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.406839 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:23.406758 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-34.us-west-2.compute.internal\": node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:47:23.438656 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.438632 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.440846 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.440946 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.440946 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440901 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.440946 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.440946 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440923 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.441092 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.440952 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.498270 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.498202 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:47:23.498388 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.498317 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.557199 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.557174 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:47:23.656678 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.656649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:23.659299 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:23.659274 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:47:25.299871 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.299631 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:47:25.300345 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.299918 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:47:25.300345 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.299965 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:47:25.300345 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.299983 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:47:25.300345 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.300009 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a494c12af1f134ca494febab804ea946-87fdf5767f91ce37.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-34.us-west-2.compute.internal.18e4521e9e450a84\": read tcp 10.0.128.34:60826->3.132.191.132:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-34.us-west-2.compute.internal.18e4521e9e450a84 default 5211 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-34.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-34.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-34.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:23 +0000 UTC,LastTimestamp:2026-11-02 19:47:23.305620001 +0000 UTC m=+0.692862641,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-34.us-west-2.compute.internal,}" Nov 02 19:47:25.340823 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.340797 2581 apiserver.go:52] "Watching apiserver" Nov 02 19:47:25.365709 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.365687 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:47:25.580827 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.580741 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:24 +0000 UTC" deadline="2028-06-11 15:18:08.395726029 +0000 UTC" Nov 02 19:47:25.580827 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.580772 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14083h30m42.814957258s" Nov 02 19:47:25.622436 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.622412 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:47:25.622773 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.622754 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-nvcwl","openshift-cluster-node-tuning-operator/tuned-hsvmf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal","openshift-multus/multus-2jzk4","openshift-network-operator/iptables-alerter-9hdxl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl","openshift-dns/node-resolver-zpmbp","openshift-image-registry/node-ca-5s9lr","openshift-multus/network-metrics-daemon-tf5w7","openshift-multus/multus-additional-cni-plugins-xwh2b","openshift-ovn-kubernetes/ovnkube-node-bff7p","kube-system/global-pull-secret-syncer-h6fsm","kube-system/konnectivity-agent-wfdgc"] Nov 02 19:47:25.624343 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.624329 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:25.624407 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.624392 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:25.625423 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.625409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.626772 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.626752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.628007 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.627990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.629306 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.629290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.630442 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.630425 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.632125 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.632109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.633472 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.633454 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.633552 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.633504 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:25.634789 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.634775 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.636271 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.636254 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.637539 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.637521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.637624 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.637569 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:25.638922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.638904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.668785 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.668770 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:47:25.676310 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.676393 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.676451 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.676508 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676466 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.676582 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.676634 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676590 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.676634 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676606 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.676634 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.676807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.676807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.676807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.676916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.676916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.676916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrx5m\" (UniqueName: \"kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.677026 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.677026 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677026 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:25.677026 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.676973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.677199 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.677199 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677061 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.677199 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfkj\" (UniqueName: \"kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.677199 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.677332 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677332 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677332 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtvbj\" (UniqueName: \"kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677485 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677564 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffkl\" (UniqueName: \"kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.677922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.677969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.678293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mvd\" (UniqueName: \"kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9glh\" (UniqueName: \"kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd22g\" (UniqueName: \"kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ljq\" (UniqueName: \"kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.678640 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspt9\" (UniqueName: \"kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678941 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.678979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.679149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.679596 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679188 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.679596 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.679596 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.679247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.680274 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.680271 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.680346 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.681340 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.681358 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.681425 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.681479 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.681441 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:47:25.681863 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.681489 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:47:25.682150 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.682128 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.682219 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.682203 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:47:25.682274 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.682260 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.682508 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.682491 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\"" Nov 02 19:47:25.683170 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.683142 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.683579 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.683558 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.683687 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.683612 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:47:25.683687 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.683652 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\"" Nov 02 19:47:25.684134 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.684119 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.684308 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.684293 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.684428 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.684415 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\"" Nov 02 19:47:25.776721 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.776698 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:47:25.776832 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.776771 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f5cw8" Nov 02 19:47:25.779922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.779903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.779999 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.779928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.779999 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.779947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.779999 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.779961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.779999 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.779984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780008 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780052 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780090 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.780101 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780143 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mvd\" (UniqueName: \"kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.780186 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.780188 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:26.280165457 +0000 UTC m=+2.338534245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780179 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780210 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780259 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9glh\" (UniqueName: \"kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd22g\" (UniqueName: \"kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780379 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ljq\" (UniqueName: \"kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780465 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780471 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.780430 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:25.780742 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.780529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:26.280513267 +0000 UTC m=+2.338882058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780531 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780475 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780554 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780619 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspt9\" (UniqueName: \"kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780749 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.781448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780804 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780822 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780887 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780925 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780942 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.780943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781085 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781091 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.782206 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781189 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781250 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781276 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781295 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781310 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.783043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrx5m\" (UniqueName: \"kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781372 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781424 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781459 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781470 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781511 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781569 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781255 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781625 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfkj\" (UniqueName: \"kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781708 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\"" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781716 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.783963 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781770 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781790 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781808 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781822 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\"" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781842 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtvbj\" (UniqueName: \"kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781883 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781912 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781952 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.784526 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781994 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781999 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782047 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkl\" (UniqueName: \"kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782249 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782553 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.785185 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782666 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782673 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.782983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783050 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783096 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783266 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783326 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.781947 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783608 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783653 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.783791 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.784319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.786039 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.785788 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.786949 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.785850 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.786949 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.786108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.791980 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.791958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:25.792178 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.792162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.793343 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.793325 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.842978 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.842913 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mvd\" (UniqueName: \"kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.843263 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.843245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd22g\" (UniqueName: \"kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:25.843922 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.843903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9glh\" (UniqueName: \"kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:25.849791 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.849770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspt9\" (UniqueName: \"kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.852565 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.852538 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ljq\" (UniqueName: \"kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:25.852664 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.852628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffkl\" (UniqueName: \"kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:25.852842 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.852823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfkj\" (UniqueName: \"kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:25.853050 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.853036 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:25.853099 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.853053 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:25.853099 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.853062 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:25.853166 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:25.853110 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:26.353096123 +0000 UTC m=+2.411464899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:25.854034 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.854019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrx5m\" (UniqueName: \"kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:25.857468 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.857445 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtvbj\" (UniqueName: \"kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:25.954004 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.953978 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:47:25.980621 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.980601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jzk4" Nov 02 19:47:25.993959 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:25.993935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:47:26.014361 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.014336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:47:26.025028 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.025003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:47:26.039975 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.039954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:47:26.060037 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.059952 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:47:26.078900 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.078877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:26.081577 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.081555 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:26.093531 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.093485 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53a41f215f25804a365e963f8c63305.slice/crio-e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8 WatchSource:0}: Error finding container e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8: Status 404 returned error can't find the container with id e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8 Nov 02 19:47:26.094129 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.094108 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf3727f_40a7_4555_8aed_2cad99a3cc4d.slice/crio-ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224 WatchSource:0}: Error finding container ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224: Status 404 returned error can't find the container with id ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224 Nov 02 19:47:26.094871 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.094852 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6842654_0249_4e3b_ade8_d0e46997bed4.slice/crio-2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68 WatchSource:0}: Error finding container 2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68: Status 404 returned error can't find the container with id 2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68 Nov 02 19:47:26.095622 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.095602 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc6ff8a97420718ecc4211b6b597a48.slice/crio-80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8 WatchSource:0}: Error finding container 80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8: Status 404 returned error can't find the container with id 80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8 Nov 02 19:47:26.096072 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.096054 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ba6228_6606_4b2b_a7b0_04f13a9e7e0c.slice/crio-8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf WatchSource:0}: Error finding container 8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf: Status 404 returned error can't find the container with id 8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf Nov 02 19:47:26.099409 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.099332 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:47:26.285525 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.285496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:26.285692 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.285534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:26.285692 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.285628 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:26.285692 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.285630 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:26.285692 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.285675 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.285663686 +0000 UTC m=+3.344032462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:26.285692 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.285687 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.285681572 +0000 UTC m=+3.344050348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:26.374227 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.374158 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:26.386651 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.386624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:26.386810 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.386795 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:26.386885 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.386818 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:26.386885 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.386830 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:26.386961 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:26.386889 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:27.386871349 +0000 UTC m=+3.445240142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:26.536937 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.536891 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf"} Nov 02 19:47:26.537908 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.537887 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerStarted","Data":"80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8"} Nov 02 19:47:26.538916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.538895 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wfdgc" event={"ID":"b6842654-0249-4e3b-ade8-d0e46997bed4","Type":"ContainerStarted","Data":"2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68"} Nov 02 19:47:26.539876 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.539857 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"d53a41f215f25804a365e963f8c63305","Type":"ContainerStarted","Data":"e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8"} Nov 02 19:47:26.540898 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.540878 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerStarted","Data":"ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224"} Nov 02 19:47:26.769491 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.769425 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:26.778779 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.778726 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:25 +0000 UTC" deadline="2028-04-18 14:30:47.299218578 +0000 UTC" Nov 02 19:47:26.778779 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.778778 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12786h43m20.52044266s" Nov 02 19:47:26.788066 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.788035 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad4cb29_a4bf_4374_8422_f9b42da120fb.slice/crio-5ea01f5eebf8bfc4480e84ae37b270858be7598b8d491b4cb0f4e7c79ac17c6a WatchSource:0}: Error finding container 5ea01f5eebf8bfc4480e84ae37b270858be7598b8d491b4cb0f4e7c79ac17c6a: Status 404 returned error can't find the container with id 5ea01f5eebf8bfc4480e84ae37b270858be7598b8d491b4cb0f4e7c79ac17c6a Nov 02 19:47:26.795797 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.795775 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2e53d6_22a9_4364_a668_d09a984ea64f.slice/crio-55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14 WatchSource:0}: Error finding container 55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14: Status 404 returned error can't find the container with id 55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14 Nov 02 19:47:26.811922 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.811898 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce2cd6d_38ca_4f4f_87fa_0b12f8bef4d4.slice/crio-e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c WatchSource:0}: Error finding container e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c: Status 404 returned error can't find the container with id e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c Nov 02 19:47:26.825156 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:26.825139 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:26.869007 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.868978 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4588a7b1_e4b1_4255_bf39_d51481fb034f.slice/crio-fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf WatchSource:0}: Error finding container fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf: Status 404 returned error can't find the container with id fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf Nov 02 19:47:26.899504 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.899481 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede0288a_b302_459e_9eaf_8742d594edf3.slice/crio-6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f WatchSource:0}: Error finding container 6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f: Status 404 returned error can't find the container with id 6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f Nov 02 19:47:26.900032 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:47:26.900009 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d12f6a_8088_4c74_b3d5_7a8867a13872.slice/crio-04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5 WatchSource:0}: Error finding container 04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5: Status 404 returned error can't find the container with id 04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5 Nov 02 19:47:27.293368 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.293334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:27.293545 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.293387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:27.293644 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.293627 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:27.293701 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.293692 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:29.293673372 +0000 UTC m=+5.352042163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:27.293830 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.293792 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:27.293830 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.293828 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:29.293816147 +0000 UTC m=+5.352184938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:27.394508 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.394474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:27.395095 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.394630 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:27.395095 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.394648 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:27.395095 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.394660 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:27.395095 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.394715 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:29.394696198 +0000 UTC m=+5.453064989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:27.533033 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.533001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:27.533210 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.533146 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:27.533276 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.533229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:27.533314 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.533287 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:27.533352 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.533329 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:27.533385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:27.533367 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:27.548835 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.548695 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jzk4" event={"ID":"51d12f6a-8088-4c74-b3d5-7a8867a13872","Type":"ContainerStarted","Data":"04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5"} Nov 02 19:47:27.552262 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.552171 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" event={"ID":"ede0288a-b302-459e-9eaf-8742d594edf3","Type":"ContainerStarted","Data":"6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f"} Nov 02 19:47:27.554825 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.554761 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5s9lr" event={"ID":"ff2e53d6-22a9-4364-a668-d09a984ea64f","Type":"ContainerStarted","Data":"55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14"} Nov 02 19:47:27.559487 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.559401 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9hdxl" event={"ID":"4588a7b1-e4b1-4255-bf39-d51481fb034f","Type":"ContainerStarted","Data":"fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf"} Nov 02 19:47:27.561764 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.561671 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpmbp" event={"ID":"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4","Type":"ContainerStarted","Data":"e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c"} Nov 02 19:47:27.563784 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.563707 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" event={"ID":"0ad4cb29-a4bf-4374-8422-f9b42da120fb","Type":"ContainerStarted","Data":"5ea01f5eebf8bfc4480e84ae37b270858be7598b8d491b4cb0f4e7c79ac17c6a"} Nov 02 19:47:27.779337 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.779225 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:25 +0000 UTC" deadline="2028-06-18 17:19:18.475969437 +0000 UTC" Nov 02 19:47:27.779337 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:27.779288 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14253h31m50.696686068s" Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.315695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.315766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.315914 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.315979 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:33.315961878 +0000 UTC m=+9.374330657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.316681 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:29.316791 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.316755 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:33.316720719 +0000 UTC m=+9.375089496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:29.417471 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.417145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:29.417471 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.417338 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:29.417471 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.417360 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:29.417471 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.417372 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:29.417471 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.417427 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:33.417409929 +0000 UTC m=+9.475778706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:29.533495 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.533466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:29.533649 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.533585 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:29.533724 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.533651 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:29.533805 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.533760 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:29.533863 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.533809 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:29.533914 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:29.533877 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:29.600302 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:29.600265 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerStarted","Data":"e1838825d5d683ea6cac2c0deeb657fb26bdefe5b946013b385a0bb340aded1d"} Nov 02 19:47:30.617712 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.617457 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wfdgc" event={"ID":"b6842654-0249-4e3b-ade8-d0e46997bed4","Type":"ContainerStarted","Data":"7e7bc265687b2ade17bd3363b25b493667eea6de72fe93a56a98941e1773d21e"} Nov 02 19:47:30.625481 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.625448 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="e1838825d5d683ea6cac2c0deeb657fb26bdefe5b946013b385a0bb340aded1d" exitCode=0 Nov 02 19:47:30.625617 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.625535 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"e1838825d5d683ea6cac2c0deeb657fb26bdefe5b946013b385a0bb340aded1d"} Nov 02 19:47:30.643051 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.642495 2581 generic.go:356] "Generic (PLEG): container finished" podID="fbc6ff8a97420718ecc4211b6b597a48" containerID="48d6f39bc9840d5f2c2916ba00ce215e2c7df27c47a54e7bcae33b9e9660408c" exitCode=0 Nov 02 19:47:30.643051 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.642548 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerDied","Data":"48d6f39bc9840d5f2c2916ba00ce215e2c7df27c47a54e7bcae33b9e9660408c"} Nov 02 19:47:30.860353 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:30.860291 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wfdgc" podStartSLOduration=2.625475583 podStartE2EDuration="6.860271721s" totalImagesPullingTime="4.234796138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:30.736205732 +0000 UTC m=+6.794574531" watchObservedRunningTime="2026-11-02 19:47:30.860271721 +0000 UTC m=+6.918640520" Nov 02 19:47:31.533268 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.533236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:31.533441 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:31.533355 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:31.533513 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.533444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:31.533563 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:31.533524 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:31.533616 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.533578 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:31.533665 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:31.533628 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:31.648590 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.648023 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerStarted","Data":"4c71b35a86eb4809d0999a4b2af083adf53fe0cea0fb9afb141c0c4fabf481fb"} Nov 02 19:47:31.652285 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.652207 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"d53a41f215f25804a365e963f8c63305","Type":"ContainerStarted","Data":"04f53fbad0faecfff8b96e7b20d8d323689495685fa20d2ccd7e55e0e5d1d2d2"} Nov 02 19:47:31.886313 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:31.886267 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" podStartSLOduration=7.886252676 podStartE2EDuration="7.886252676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:31.769758627 +0000 UTC m=+7.828127425" watchObservedRunningTime="2026-11-02 19:47:31.886252676 +0000 UTC m=+7.944621474" Nov 02 19:47:32.664001 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:32.663853 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5s9lr" event={"ID":"ff2e53d6-22a9-4364-a668-d09a984ea64f","Type":"ContainerStarted","Data":"d5750a5d766230636a8dae2defe441216902feff3ea575d9c3ddd8ab37387548"} Nov 02 19:47:32.781916 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:32.781863 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" podStartSLOduration=8.781843615 podStartE2EDuration="8.781843615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:31.886982451 +0000 UTC m=+7.945351251" watchObservedRunningTime="2026-11-02 19:47:32.781843615 +0000 UTC m=+8.840212412" Nov 02 19:47:32.873097 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:32.873017 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:32.873777 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:32.873752 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:32.998286 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:32.998199 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5s9lr" podStartSLOduration=3.929277357 podStartE2EDuration="8.99817929s" totalImagesPullingTime="5.068901933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:32.782908183 +0000 UTC m=+8.841276981" watchObservedRunningTime="2026-11-02 19:47:32.99817929 +0000 UTC m=+9.056548090" Nov 02 19:47:33.350364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.350294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:33.350364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.350329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:33.350585 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.350444 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:33.350585 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.350498 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.350479524 +0000 UTC m=+17.408848300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:33.350865 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.350844 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:33.350951 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.350898 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.350883033 +0000 UTC m=+17.409251808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:33.450969 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.450917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:33.451145 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.451127 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:33.451210 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.451151 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:33.451210 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.451163 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:33.451299 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.451222 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.451205212 +0000 UTC m=+17.509573993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:33.533271 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.533240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:33.533271 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.533269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:33.533548 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.533383 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:33.533548 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.533450 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:33.533548 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.533501 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:33.533681 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:33.533572 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:33.669032 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:33.668558 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" event={"ID":"ede0288a-b302-459e-9eaf-8742d594edf3","Type":"ContainerStarted","Data":"2ba409fc90fca471ff9613f49a3cd86601ddef4f1d86e14bae88935b0d77a8f8"} Nov 02 19:47:34.659883 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.659840 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:47:34.673048 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.672343 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpmbp" event={"ID":"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4","Type":"ContainerStarted","Data":"f27901e84b2dd4ee1f171851a4dbd392d516eb47187f631783d19ab3dbe3232e"} Nov 02 19:47:34.674813 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.674745 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jzk4" event={"ID":"51d12f6a-8088-4c74-b3d5-7a8867a13872","Type":"ContainerStarted","Data":"e65d8d55b44e6f2713d79919ab43a98dab633af5650de39eed087557a32a1506"} Nov 02 19:47:34.676508 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.676487 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" event={"ID":"ede0288a-b302-459e-9eaf-8742d594edf3","Type":"ContainerStarted","Data":"e7142642d101bc695092f8c9cb7597d0d3bb5d9c4de835019a10152fbefeb315"} Nov 02 19:47:34.790349 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.790291 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zpmbp" podStartSLOduration=3.428635093 podStartE2EDuration="10.79027047s" totalImagesPullingTime="7.361635377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:34.789764194 +0000 UTC m=+10.848132993" watchObservedRunningTime="2026-11-02 19:47:34.79027047 +0000 UTC m=+10.848639298" Nov 02 19:47:34.908782 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:34.908691 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2jzk4" podStartSLOduration=3.698736674 podStartE2EDuration="10.908677859s" totalImagesPullingTime="7.209941185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:34.908630032 +0000 UTC m=+10.966998839" watchObservedRunningTime="2026-11-02 19:47:34.908677859 +0000 UTC m=+10.967046656" Nov 02 19:47:35.456239 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.456126 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:47:34.659861053Z","UUID":"2d5323c7-8211-442a-bb13-1f2d826f643f","Handler":null,"Name":"","Endpoint":""} Nov 02 19:47:35.458195 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.458175 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:47:35.458311 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.458201 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:47:35.533677 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.533648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:35.533677 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.533677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:35.534091 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:35.533754 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:35.534091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.533782 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:35.534091 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:35.533866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:35.534091 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:35.533961 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:35.692448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.692414 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="3a949bee168830b6e42394e8a48f634e1294a9b6afa80eb3c0895e6666f0838e" exitCode=0 Nov 02 19:47:35.692828 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:35.692551 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"3a949bee168830b6e42394e8a48f634e1294a9b6afa80eb3c0895e6666f0838e"} Nov 02 19:47:36.696989 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:36.696954 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" event={"ID":"ede0288a-b302-459e-9eaf-8742d594edf3","Type":"ContainerStarted","Data":"cf42e921c26c7f6e9d323c5fc6485302ac5a4fe3d8b1eb0d2d32309dc7c0c40d"} Nov 02 19:47:36.699099 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:36.699067 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9hdxl" event={"ID":"4588a7b1-e4b1-4255-bf39-d51481fb034f","Type":"ContainerStarted","Data":"9ac726b9734071617acc5f12e01437c86737cd206bc0ab60db0195e94339d6f2"} Nov 02 19:47:36.935044 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:36.934871 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9hdxl" podStartSLOduration=4.827027889 podStartE2EDuration="12.934855303s" totalImagesPullingTime="8.107827414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.934367354 +0000 UTC m=+12.992736153" watchObservedRunningTime="2026-11-02 19:47:36.934855303 +0000 UTC m=+12.993224106" Nov 02 19:47:36.935153 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:36.935081 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" podStartSLOduration=4.195401495 podStartE2EDuration="12.935075367s" totalImagesPullingTime="8.739673872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.81791595 +0000 UTC m=+12.876284748" watchObservedRunningTime="2026-11-02 19:47:36.935075367 +0000 UTC m=+12.993444166" Nov 02 19:47:37.533723 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:37.533664 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:37.533917 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:37.533749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:37.533917 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:37.533841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:37.533917 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:37.533857 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:37.534069 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:37.533929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:37.534069 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:37.534019 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:37.702556 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:37.702518 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="cd8235ecc9ff79e1e71a2c5c1c35ea1603fbef0270968a25f62820d99d3b9a1d" exitCode=0 Nov 02 19:47:37.702556 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:37.702545 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"cd8235ecc9ff79e1e71a2c5c1c35ea1603fbef0270968a25f62820d99d3b9a1d"} Nov 02 19:47:38.705621 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:38.705582 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" event={"ID":"0ad4cb29-a4bf-4374-8422-f9b42da120fb","Type":"ContainerStarted","Data":"6c930875e5748dee86aff39e9f5f8ffc5be37b4b570a154ce25a813f9d6fa0b3"} Nov 02 19:47:38.824975 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:38.824919 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" podStartSLOduration=3.46851834 podStartE2EDuration="14.824903599s" totalImagesPullingTime="11.356385259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:38.824092359 +0000 UTC m=+14.882461158" watchObservedRunningTime="2026-11-02 19:47:38.824903599 +0000 UTC m=+14.883272397" Nov 02 19:47:39.533569 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:39.533541 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:39.533703 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:39.533602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:39.533703 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:39.533691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:39.533805 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:39.533704 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:39.533805 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:39.533787 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:39.533887 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:39.533866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:39.710827 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:39.710780 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="35ae59d244970345dcb6f7c1e1570777855304825987387d54f0fa629c83b406" exitCode=0 Nov 02 19:47:39.711497 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:39.710887 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"35ae59d244970345dcb6f7c1e1570777855304825987387d54f0fa629c83b406"} Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.415896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.416173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.416185 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.416253 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:47:57.416234371 +0000 UTC m=+33.474603163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.416313 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:41.416635 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.416373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:57.416355444 +0000 UTC m=+33.474724222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:41.516655 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.516617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:41.516840 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.516754 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:41.516840 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.516772 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:41.516840 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.516782 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:41.516840 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.516833 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:57.516816318 +0000 UTC m=+33.575185109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:41.533493 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.533460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:41.533622 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.533501 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:41.533675 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.533625 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:41.533675 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:41.533669 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:41.533788 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.533772 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:41.533856 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:41.533839 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:42.719762 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:42.719716 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="249191cb96d933209ffe62b75633e6a66f0a776f0d75199003c9049589a5570a" exitCode=0 Nov 02 19:47:42.720096 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:42.719762 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"249191cb96d933209ffe62b75633e6a66f0a776f0d75199003c9049589a5570a"} Nov 02 19:47:43.532961 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:43.532930 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:43.533127 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:43.532928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:43.533127 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:43.533053 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:43.533127 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:43.533099 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:43.533127 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:43.532943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:43.533329 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:43.533171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:43.725129 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:43.725092 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="3c35d5dd913040ca819262384f7f7c3763d9f361db8e6dfef98c1dda81c0fe38" exitCode=0 Nov 02 19:47:43.725508 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:43.725143 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerDied","Data":"3c35d5dd913040ca819262384f7f7c3763d9f361db8e6dfef98c1dda81c0fe38"} Nov 02 19:47:44.730957 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:44.730920 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" event={"ID":"faf3727f-40a7-4555-8aed-2cad99a3cc4d","Type":"ContainerStarted","Data":"522fd06bf06cf3a33731d209ef45e86e150948f3fd6c0ed80c8fc3ff418c87a5"} Nov 02 19:47:44.957245 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:44.957199 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xwh2b" podStartSLOduration=8.808647426 podStartE2EDuration="20.95718677s" totalImagesPullingTime="12.148539344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:44.904647444 +0000 UTC m=+20.963016242" watchObservedRunningTime="2026-11-02 19:47:44.95718677 +0000 UTC m=+21.015555566" Nov 02 19:47:45.533171 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:45.533129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:45.533416 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:45.533192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:45.533416 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:45.533199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:45.533416 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:45.533289 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:45.533565 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:45.533427 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:45.533565 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:45.533518 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:47.533052 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:47.533015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:47.533583 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:47.533057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:47.533583 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:47.533057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:47.533583 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:47.533131 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:47.533583 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:47.533255 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:47.533583 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:47.533343 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:48.743540 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:48.743371 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"7091ed4fccf2abd04cbb5a5fb9dfe3594ee4179c6a223a4577eb6d682db45c6d"} Nov 02 19:47:48.743852 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:48.743583 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"4936ce031ba29c86f7037ac8c9431a95ca63fe37e0155f3909d7f2c13fc7201f"} Nov 02 19:47:48.743852 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:48.743614 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"dec2956a39d762de6539a2d2722941d85a86922c151eea5d1ccfc011f5352f18"} Nov 02 19:47:48.743852 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:48.743641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"ac5a0a85670260d37ede47842884be7d0b419c9bbe5a14f281cbad27bcaa3e30"} Nov 02 19:47:49.533075 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:49.533036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:49.533075 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:49.533061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:49.533274 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:49.533153 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:49.533274 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:49.533204 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:49.533340 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:49.533290 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:49.533374 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:49.533360 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:49.747851 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:49.747814 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"c0c0d2488ae47f0a65a88d9d9a5aa310a797ce1a59be45825e71cf477147a90e"} Nov 02 19:47:49.748227 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:49.747880 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"3134e2807f0da0898fbf6c8b392d1790b694870b644d45272f9fc5c97e4cb0ff"} Nov 02 19:47:51.223379 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.223294 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:51.223807 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.223417 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:47:51.223876 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.223858 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:47:51.533794 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.533710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:51.533930 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.533801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:51.533930 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.533889 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:51.533930 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:51.533883 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:51.534026 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:51.533960 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:51.534057 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:51.534021 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:51.755903 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:51.755870 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"f46e726d8e8cab2aab38515cbc79d25be8675747a4709074b3478eb444b0f442"} Nov 02 19:47:53.533480 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.533445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:53.533480 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.533477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:53.534263 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.533484 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:53.534263 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:53.533555 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:53.534263 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:53.533653 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:53.534263 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:53.533751 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:53.763313 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.763282 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" event={"ID":"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c","Type":"ContainerStarted","Data":"9d6675e79495a96f16294527811f169266af6a55f93a6c0dca31145bfec94731"} Nov 02 19:47:53.763502 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.763486 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:53.778301 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:53.778279 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:54.047265 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:54.047220 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" podStartSLOduration=7.670584994 podStartE2EDuration="30.0472071s" totalImagesPullingTime="22.376622106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:54.046567218 +0000 UTC m=+30.104936015" watchObservedRunningTime="2026-11-02 19:47:54.0472071 +0000 UTC m=+30.105575898" Nov 02 19:47:54.765962 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:54.765931 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:54.766384 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:54.766000 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:54.781417 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:54.781392 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:47:55.533719 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:55.533690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:55.533902 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:55.533766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:55.533902 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:55.533850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:55.533902 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:55.533849 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:55.534012 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:55.533907 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:55.534012 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:55.533990 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:56.437214 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:56.437186 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpmbp_5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4/dns-node-resolver/0.log" Nov 02 19:47:56.865997 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:56.865927 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5s9lr_ff2e53d6-22a9-4364-a668-d09a984ea64f/node-ca/0.log" Nov 02 19:47:57.424675 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.424643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:57.424846 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.424683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:57.424846 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.424803 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:57.424933 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.424859 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs podName:66d713cc-a097-4c92-bafe-e899ece7bc54 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:29.424846664 +0000 UTC m=+65.483215440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs") pod "network-metrics-daemon-tf5w7" (UID: "66d713cc-a097-4c92-bafe-e899ece7bc54") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:57.424933 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.424804 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:57.425009 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.424939 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret podName:e904a0d4-e8e9-4279-bba7-212e6b10fb5d nodeName:}" failed. No retries permitted until 2026-11-02 19:48:29.424923074 +0000 UTC m=+65.483291851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret") pod "global-pull-secret-syncer-h6fsm" (UID: "e904a0d4-e8e9-4279-bba7-212e6b10fb5d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:57.525425 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.525387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:57.525800 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.525533 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:57.525800 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.525553 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:57.525800 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.525563 2581 projected.go:196] Error preparing data for projected volume kube-api-access-l755k for pod openshift-network-diagnostics/network-check-target-nvcwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:57.525800 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.525606 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k podName:62352636-2385-40c9-8beb-6889208afd98 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:29.525593264 +0000 UTC m=+65.583962041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l755k" (UniqueName: "kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k") pod "network-check-target-nvcwl" (UID: "62352636-2385-40c9-8beb-6889208afd98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:57.532810 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.532792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:57.532884 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.532836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:57.532884 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:57.532839 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:57.532950 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.532900 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:57.532997 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.532977 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:47:57.533071 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:57.533050 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:59.533114 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:59.533082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:47:59.533114 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:59.533099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:47:59.533553 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:47:59.533151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:47:59.533553 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:59.533235 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:47:59.533553 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:59.533347 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:47:59.533553 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:47:59.533399 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:01.533616 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:01.533590 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:01.533616 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:01.533618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:01.534108 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:01.533593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:01.534108 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:01.533677 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:01.534108 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:01.533781 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:01.534108 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:01.533884 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:03.533632 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:03.533600 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:03.534149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:03.533739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:03.534149 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:03.533772 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:03.534149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:03.533814 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:03.534149 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:03.533909 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:03.534149 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:03.533990 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:05.533383 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:05.533209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:05.533833 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:05.533209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:05.533833 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:05.533454 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:05.533833 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:05.533536 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:05.533833 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:05.533209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:05.533833 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:05.533614 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:07.532925 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:07.532888 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:07.532925 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:07.532919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:07.532925 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:07.532932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:07.533385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:07.533012 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:07.533385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:07.533065 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:07.533385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:07.533140 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:09.533018 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:09.532983 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:09.533492 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:09.532983 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:09.533492 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:09.533116 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:09.533492 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:09.532983 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:09.533492 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:09.533159 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:09.533492 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:09.533222 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:11.533716 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:11.533681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:11.534094 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:11.533798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:11.534094 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:11.533809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:11.534094 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:11.533825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:11.534094 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:11.533906 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:11.534094 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:11.533977 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:13.533457 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.533419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:13.533457 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.533448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:13.533935 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.533448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:13.533935 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.533530 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:13.533935 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.533667 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:13.533935 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.533706 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:13.561131 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.561104 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h6fsm"] Nov 02 19:48:13.563629 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.563590 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nvcwl"] Nov 02 19:48:13.564107 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.564090 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tf5w7"] Nov 02 19:48:13.801215 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.801146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:13.801385 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.801146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:13.801385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.801256 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:13.801385 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:13.801146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:13.801385 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.801330 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:13.801515 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:13.801383 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:15.533171 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:15.533138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:15.533665 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:15.533173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:15.533665 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:15.533229 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:15.533665 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:15.533271 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:15.533665 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:15.533368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:15.533665 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:15.533400 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:17.533775 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:17.533715 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:17.534198 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:17.533848 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tf5w7" podUID="66d713cc-a097-4c92-bafe-e899ece7bc54" Nov 02 19:48:17.534198 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:17.533758 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:17.534198 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:17.533749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:17.534198 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:17.533916 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6fsm" podUID="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" Nov 02 19:48:17.534198 ip-10-0-128-34 kubenswrapper[2581]: E1102 19:48:17.533992 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvcwl" podUID="62352636-2385-40c9-8beb-6889208afd98" Nov 02 19:48:18.211981 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.211957 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeReady" Nov 02 19:48:18.212129 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.212088 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:48:18.373964 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.373917 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hspfn"] Nov 02 19:48:18.376523 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.376497 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jwdhv"] Nov 02 19:48:18.376629 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.376615 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.378558 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.378539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.381142 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.381126 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hspfn"] Nov 02 19:48:18.381641 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.381625 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jwdhv"] Nov 02 19:48:18.405292 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.405271 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nxh58"] Nov 02 19:48:18.407299 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.407285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.432043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432003 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:48:18.432043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432042 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:48:18.432043 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432015 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:48:18.432241 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432093 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\"" Nov 02 19:48:18.432241 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432166 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:48:18.432943 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.432927 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:48:18.433035 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.433022 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:48:18.435150 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.435139 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:48:18.438274 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.438257 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\"" Nov 02 19:48:18.463364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.463319 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxh58"] Nov 02 19:48:18.480111 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtp2\" (UniqueName: \"kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.480229 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.480229 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.480297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.480297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.480297 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480287 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvr2x\" (UniqueName: \"kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.480385 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.480385 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.480444 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.480474 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.480505 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.480484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.533293 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.533262 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:48:18.533410 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.533302 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:48:18.533410 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.533305 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\"" Nov 02 19:48:18.581517 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.581517 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvr2x\" (UniqueName: \"kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581709 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.581990 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtp2\" (UniqueName: \"kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.582358 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.581996 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.582358 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.582007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.582358 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.582278 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.582449 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.582359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.582669 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.582645 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.585191 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.585170 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.585284 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.585271 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.585399 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.585328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.641610 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.641581 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtp2\" (UniqueName: \"kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:18.641793 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.641775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.642132 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.642116 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvr2x\" (UniqueName: \"kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.691341 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.691312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:18.693470 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.693449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:18.717372 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:18.717311 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:19.073839 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:19.073803 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfa771b_a624_46be_8b7a_869cc29cb8c4.slice/crio-c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e WatchSource:0}: Error finding container c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e: Status 404 returned error can't find the container with id c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e Nov 02 19:48:19.075761 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.075715 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jwdhv"] Nov 02 19:48:19.121267 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:19.121235 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac4ccdd_a8e0_484d_acb2_fa7fddc0da3c.slice/crio-5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47 WatchSource:0}: Error finding container 5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47: Status 404 returned error can't find the container with id 5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47 Nov 02 19:48:19.123795 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.123771 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hspfn"] Nov 02 19:48:19.126420 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:19.126397 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64732a9_9d3a_4d5e_8b01_f5ebef4a986c.slice/crio-0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd WatchSource:0}: Error finding container 0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd: Status 404 returned error can't find the container with id 0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd Nov 02 19:48:19.126830 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.126811 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxh58"] Nov 02 19:48:19.533285 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.533254 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:19.533448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.533318 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:19.533448 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.533409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:19.588666 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588635 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:48:19.588666 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588636 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:48:19.589065 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588702 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:48:19.589065 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588707 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:48:19.589065 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588828 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:48:19.589065 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588877 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:48:19.589065 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.588876 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:48:19.815374 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.815283 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxh58" event={"ID":"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c","Type":"ContainerStarted","Data":"0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd"} Nov 02 19:48:19.816369 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.816340 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jwdhv" event={"ID":"6bfa771b-a624-46be-8b7a-869cc29cb8c4","Type":"ContainerStarted","Data":"c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e"} Nov 02 19:48:19.817667 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.817645 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hspfn" event={"ID":"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c","Type":"ContainerStarted","Data":"1cc53727a6fc2133aa37b5ce21779a34ddad9e72b4f647f066564e6d41179f63"} Nov 02 19:48:19.817769 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:19.817688 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hspfn" event={"ID":"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c","Type":"ContainerStarted","Data":"5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47"} Nov 02 19:48:20.823289 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:20.823206 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hspfn" event={"ID":"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c","Type":"ContainerStarted","Data":"283d5daa8bb94293830cb941e91b8e65727083cff3526605cb655186f11d7559"} Nov 02 19:48:22.830584 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.830498 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxh58" event={"ID":"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c","Type":"ContainerStarted","Data":"127852a2e97c922214a3213cc48d384f19eabeefb8027d456f7dbdff7ff1e928"} Nov 02 19:48:22.830584 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.830563 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxh58" event={"ID":"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c","Type":"ContainerStarted","Data":"cabca11b2cc172633ee229b51637946d8d3fa2b1130c646fbdb0f806625384d4"} Nov 02 19:48:22.831151 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.830595 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:22.832091 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.832069 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jwdhv" event={"ID":"6bfa771b-a624-46be-8b7a-869cc29cb8c4","Type":"ContainerStarted","Data":"dfdb2663bb2a5a98e496c386d3fe8f5a269f08d7d8db2d10a8b7925578a84eae"} Nov 02 19:48:22.833862 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.833844 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hspfn" event={"ID":"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c","Type":"ContainerStarted","Data":"7ac66e0ed10b3606ba58e826c2af6acb4f2cb852c39d8cbd58b7721d30bab600"} Nov 02 19:48:22.949145 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:22.949098 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nxh58" podStartSLOduration=2.124824424 podStartE2EDuration="4.949083984s" totalImagesPullingTime="2.82425956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:22.948423742 +0000 UTC m=+59.006792539" watchObservedRunningTime="2026-11-02 19:48:22.949083984 +0000 UTC m=+59.007452779" Nov 02 19:48:23.068107 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:23.068055 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hspfn" podStartSLOduration=1.803825536 podStartE2EDuration="5.068033589s" totalImagesPullingTime="3.264208053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:48:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:23.067136456 +0000 UTC m=+59.125505254" watchObservedRunningTime="2026-11-02 19:48:23.068033589 +0000 UTC m=+59.126402386" Nov 02 19:48:23.184987 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:23.184891 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jwdhv" podStartSLOduration=1.942494444 podStartE2EDuration="5.184878507s" totalImagesPullingTime="3.242384063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:23.184351151 +0000 UTC m=+59.242719949" watchObservedRunningTime="2026-11-02 19:48:23.184878507 +0000 UTC m=+59.243247305" Nov 02 19:48:26.781193 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:26.781166 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:29.456209 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.456172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:29.456209 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.456211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:29.557006 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.556968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:29.562264 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.562241 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:48:29.562323 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.562241 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:48:29.569588 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.569560 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:29.569703 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.569685 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:29.611081 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.611051 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:48:29.671542 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.671514 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:48:29.680888 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.680861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:29.797324 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.797263 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:48:29.804500 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.804472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:29.804896 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.804877 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:48:29.807597 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.807391 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:48:29.811496 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.811476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:29.814538 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:29.814524 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:30.175421 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.175346 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nvcwl"] Nov 02 19:48:30.210119 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:30.210089 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62352636_2385_40c9_8beb_6889208afd98.slice/crio-9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936 WatchSource:0}: Error finding container 9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936: Status 404 returned error can't find the container with id 9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936 Nov 02 19:48:30.218608 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:30.218586 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode904a0d4_e8e9_4279_bba7_212e6b10fb5d.slice/crio-7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525 WatchSource:0}: Error finding container 7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525: Status 404 returned error can't find the container with id 7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525 Nov 02 19:48:30.221530 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.221468 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h6fsm"] Nov 02 19:48:30.221971 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:30.221948 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d713cc_a097_4c92_bafe_e899ece7bc54.slice/crio-d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260 WatchSource:0}: Error finding container d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260: Status 404 returned error can't find the container with id d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260 Nov 02 19:48:30.229151 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.229134 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tf5w7"] Nov 02 19:48:30.852363 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.852326 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h6fsm" event={"ID":"e904a0d4-e8e9-4279-bba7-212e6b10fb5d","Type":"ContainerStarted","Data":"7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525"} Nov 02 19:48:30.853499 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.853474 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tf5w7" event={"ID":"66d713cc-a097-4c92-bafe-e899ece7bc54","Type":"ContainerStarted","Data":"d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260"} Nov 02 19:48:30.854539 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:30.854517 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nvcwl" event={"ID":"62352636-2385-40c9-8beb-6889208afd98","Type":"ContainerStarted","Data":"9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936"} Nov 02 19:48:31.380805 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.380772 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc"] Nov 02 19:48:31.383330 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.383303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.433562 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.433527 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xb77n"] Nov 02 19:48:31.436401 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.436371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.437225 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.437202 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc"] Nov 02 19:48:31.470429 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.470566 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470566 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470566 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470566 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46tt\" (UniqueName: \"kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470724 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.470783 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470767 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlv64\" (UniqueName: \"kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.471079 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.470795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.484976 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.484951 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 19:48:31.484976 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.484955 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:48:31.484976 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.484959 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k8mpj\"" Nov 02 19:48:31.485135 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.485017 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:48:31.488724 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.488705 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:48:31.488838 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.488810 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:48:31.508347 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.508315 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:48:31.537920 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.537886 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:48:31.538232 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.538208 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8hmw\"" Nov 02 19:48:31.538336 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.538298 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:48:31.572291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.572537 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46tt\" (UniqueName: \"kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.572625 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.572719 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572661 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.572719 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.572914 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.573001 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.572981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573054 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlv64\" (UniqueName: \"kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.573149 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.573267 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573229 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.573325 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573301 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573383 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573383 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573474 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573540 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573519 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.573611 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573541 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.573839 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.573810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.574051 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.574030 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.574114 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.574089 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.574201 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.574161 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.576666 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.576639 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.577097 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.577071 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.577438 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.577400 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.577616 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.577593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.632358 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.632300 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46tt\" (UniqueName: \"kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.632505 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.632485 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlv64\" (UniqueName: \"kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.700323 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.700276 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:31.700460 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.700338 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h"] Nov 02 19:48:31.703605 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.703357 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.758829 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.758795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:31.759714 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.759691 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h"] Nov 02 19:48:31.761250 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.761013 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:48:31.761250 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.761064 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:48:31.761250 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.761223 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dzphr\"" Nov 02 19:48:31.761250 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.761229 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:48:31.773599 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:31.773573 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6c4c6f_add0_4e41_a3aa_c8a48789a1a2.slice/crio-7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa WatchSource:0}: Error finding container 7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa: Status 404 returned error can't find the container with id 7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775043 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txk7x\" (UniqueName: \"kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.775291 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.775233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.859814 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.859771 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xb77n" event={"ID":"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2","Type":"ContainerStarted","Data":"7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa"} Nov 02 19:48:31.876230 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876361 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876361 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876361 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876313 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk7x\" (UniqueName: \"kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876519 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876519 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.876967 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.876939 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.877140 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.877122 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.877309 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.877285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.879019 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.878995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.879094 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.879058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:31.935544 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:31.935470 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk7x\" (UniqueName: \"kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:32.028567 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.028319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:32.118086 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:32.118056 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d90c8b2_351c_45f5_a22c_84a64a1a9fd4.slice/crio-74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb WatchSource:0}: Error finding container 74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb: Status 404 returned error can't find the container with id 74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb Nov 02 19:48:32.120173 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.119990 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc"] Nov 02 19:48:32.403194 ip-10-0-128-34 kubenswrapper[2581]: W1102 19:48:32.402700 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fb1ec8_149f_4d4b_a5b6_728048a4a0e2.slice/crio-67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949 WatchSource:0}: Error finding container 67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949: Status 404 returned error can't find the container with id 67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949 Nov 02 19:48:32.405237 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.405211 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h"] Nov 02 19:48:32.838100 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.838033 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:32.866020 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.865983 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tf5w7" event={"ID":"66d713cc-a097-4c92-bafe-e899ece7bc54","Type":"ContainerStarted","Data":"09ae4b4fd11bfa6408d8e58c2c2794b932791d3ed10f4c805b63775af31934ef"} Nov 02 19:48:32.866435 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.866067 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tf5w7" event={"ID":"66d713cc-a097-4c92-bafe-e899ece7bc54","Type":"ContainerStarted","Data":"335bc0e41905bcb4c9969ff94bb9d08cf56c6fb7c76717e4205f6ba9e92d4104"} Nov 02 19:48:32.868198 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.868174 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" event={"ID":"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4","Type":"ContainerStarted","Data":"d52e54732a7bf86a1be09e7d682ba26b453aea68950a1ab52cc262c5c60602bc"} Nov 02 19:48:32.868328 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.868223 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" event={"ID":"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4","Type":"ContainerStarted","Data":"e88ba8acc0dbe38c0463378202a77d774771a0c7c092e2aef075f5502ae52d7c"} Nov 02 19:48:32.868328 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.868254 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" event={"ID":"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4","Type":"ContainerStarted","Data":"74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb"} Nov 02 19:48:32.869345 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:32.869323 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" event={"ID":"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2","Type":"ContainerStarted","Data":"67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949"} Nov 02 19:48:33.075512 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:33.075458 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tf5w7" podStartSLOduration=66.819972458 podStartE2EDuration="1m9.0754428s" totalImagesPullingTime="2.255470342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:33.074573825 +0000 UTC m=+69.132942624" watchObservedRunningTime="2026-11-02 19:48:33.0754428 +0000 UTC m=+69.133811600" Nov 02 19:48:33.876006 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:33.875969 2581 generic.go:356] "Generic (PLEG): container finished" podID="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" containerID="7cd8e5ae68cf64b095485ef43062ac3f484c406c5f4e114f6d53ad9a7c13d318" exitCode=0 Nov 02 19:48:33.877890 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:33.877867 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xb77n" event={"ID":"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2","Type":"ContainerDied","Data":"7cd8e5ae68cf64b095485ef43062ac3f484c406c5f4e114f6d53ad9a7c13d318"} Nov 02 19:48:34.881879 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:34.881847 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" event={"ID":"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4","Type":"ContainerStarted","Data":"b0a8e14e0763106ca9a3e0992a30cef095000e895740b9eb061928674cd33ac7"} Nov 02 19:48:34.884238 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:34.884214 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xb77n" event={"ID":"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2","Type":"ContainerStarted","Data":"ec11a16c7f7e1d38daff28e4269ef6878674377ca07f3e3771b721b9619de598"} Nov 02 19:48:34.884347 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:34.884263 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xb77n" event={"ID":"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2","Type":"ContainerStarted","Data":"81dbcb6b385641b72690e314a245997c29804a0b4a3150ed68f49eb10819b63b"} Nov 02 19:48:35.127841 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:35.127745 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xb77n" podStartSLOduration=2.396408832 podStartE2EDuration="4.127715441s" totalImagesPullingTime="1.640752405s" totalInitContainerRuntime="90.554204ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:35.127540964 +0000 UTC m=+71.185909770" watchObservedRunningTime="2026-11-02 19:48:35.127715441 +0000 UTC m=+71.186084239" Nov 02 19:48:35.128132 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:35.128104 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" podStartSLOduration=2.137574037 podStartE2EDuration="4.128095181s" totalImagesPullingTime="1.990521144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:35.006116016 +0000 UTC m=+71.064484815" watchObservedRunningTime="2026-11-02 19:48:35.128095181 +0000 UTC m=+71.186463978" Nov 02 19:48:35.892364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:35.891756 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" event={"ID":"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2","Type":"ContainerStarted","Data":"c756f97bb447f8dd7cea4de9bed11a396133981d1b47c9173b5d3c9fd1cfef6d"} Nov 02 19:48:35.892364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:35.891869 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" event={"ID":"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2","Type":"ContainerStarted","Data":"85cef06d59134c7017e5d5296075888a7c02c3e556d55bca099dd9d4296cbfc4"} Nov 02 19:48:35.892364 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:35.891898 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" event={"ID":"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2","Type":"ContainerStarted","Data":"c9f7935aafa16347f37c7b585fb9bf109968e429147019e031dda458adcf530a"} Nov 02 19:48:36.068280 ip-10-0-128-34 kubenswrapper[2581]: I1102 19:48:36.067653 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" podStartSLOduration=2.602606636 podStartE2EDuration="5.067635255s" totalImagesPullingTime="2.465028619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:36.014523377 +0000 UTC m=+72.072892176" watchObservedRunningTime="2026-11-02 19:48:36.067635255 +0000 UTC m=+72.126004058" Nov 02 19:48:36.323038 ip-10-0-128-34 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:48:36.345141 ip-10-0-128-34 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:48:36.345331 ip-10-0-128-34 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:48:36.345543 ip-10-0-128-34 systemd[1]: kubelet.service: Consumed 3.532s CPU time. Nov 02 19:48:36.355201 ip-10-0-128-34 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:48:36.431914 ip-10-0-128-34 kubenswrapper[5136]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:36.431914 ip-10-0-128-34 kubenswrapper[5136]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:48:36.431914 ip-10-0-128-34 kubenswrapper[5136]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:36.431914 ip-10-0-128-34 kubenswrapper[5136]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435251 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435338 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435342 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435345 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435348 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435350 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435353 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435356 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.435349 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435360 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435363 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435366 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435370 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435375 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435379 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435383 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435388 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435391 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435394 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435397 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435400 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435410 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435414 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435417 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435420 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435422 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435425 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435428 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.435712 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435431 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435434 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435436 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435439 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435441 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435445 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435448 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435450 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435453 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435456 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435459 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435462 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435464 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435467 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435470 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435472 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435475 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435478 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435480 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435483 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.436544 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435486 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435490 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435494 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435501 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435505 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435510 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435515 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435520 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435524 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435529 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435533 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435538 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435540 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435543 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435546 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435549 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435552 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435554 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435557 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.437309 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435560 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435563 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435566 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435568 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435571 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435574 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435576 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435580 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435582 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435585 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435588 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435590 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435594 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435599 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435603 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435609 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435613 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435620 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435625 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435629 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.438031 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435632 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435637 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435641 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435645 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435648 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435653 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435657 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435661 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435664 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435668 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435671 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435674 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435677 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435682 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435684 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435688 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435692 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435696 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435701 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.438864 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.435706 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436647 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436656 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436660 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436667 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436674 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436680 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436685 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436688 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436692 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436695 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436699 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436702 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436705 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436708 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436711 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436713 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436717 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436721 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.439557 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436726 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436746 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436750 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436753 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436757 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436761 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436765 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436770 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436773 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436776 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436779 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436781 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436785 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436788 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436792 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436796 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436800 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436805 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436809 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.440325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436814 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436819 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436824 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436830 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436834 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436838 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436845 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436850 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436854 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436858 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436862 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436868 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436872 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436877 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436882 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436886 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436890 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436895 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436899 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436904 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.441023 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436909 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436914 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436918 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436923 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436927 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436931 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436937 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436943 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436947 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436951 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436955 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436960 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436965 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436969 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436973 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436977 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436983 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436987 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436992 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.436999 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.441874 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437004 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437010 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437014 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437018 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437021 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437024 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437027 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437032 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437038 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437043 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437047 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437052 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437057 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437061 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437066 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437069 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437072 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437075 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437080 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.442703 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437083 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437085 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437088 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437091 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437093 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437097 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437101 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437106 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437110 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437115 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437782 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437792 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437796 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437799 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437802 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437806 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437810 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437815 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437821 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437825 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.443441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437830 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437835 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437838 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437841 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437843 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437846 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437849 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437851 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437854 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437857 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437860 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437865 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437870 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437875 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437880 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437884 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437887 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437891 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437894 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437896 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.444374 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437899 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437902 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437905 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437907 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437910 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437913 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437916 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437919 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437921 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437924 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437929 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437933 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437938 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437942 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437947 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437951 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437954 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437959 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437963 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437969 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.445231 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437973 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437976 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437979 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437983 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437986 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437989 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437992 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437994 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.437999 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438003 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438008 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438013 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438017 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438022 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438026 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438031 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438036 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438039 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438046 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438049 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.446148 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438052 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438055 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438058 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438062 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438066 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438071 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438075 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438080 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438084 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438088 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438092 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438096 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438101 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438106 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438110 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438114 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438117 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438120 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438122 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.447039 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438125 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438128 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438131 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438134 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438138 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438142 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438144 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438149 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438153 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438159 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438164 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438168 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438174 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438178 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438181 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438184 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.438187 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438306 5136 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438323 5136 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438333 5136 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:48:36.447970 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438339 5136 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438347 5136 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438352 5136 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438359 5136 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438366 5136 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438371 5136 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438376 5136 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438381 5136 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438385 5136 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438388 5136 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438392 5136 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438395 5136 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438398 5136 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438401 5136 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438404 5136 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438409 5136 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438412 5136 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438415 5136 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438420 5136 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438426 5136 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438432 5136 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438435 5136 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438440 5136 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:48:36.448819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438446 5136 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438453 5136 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438467 5136 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438472 5136 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438475 5136 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438478 5136 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438483 5136 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438486 5136 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438489 5136 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438493 5136 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438499 5136 flags.go:64] FLAG: --enable-server="true" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438506 5136 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438513 5136 flags.go:64] FLAG: --event-burst="100" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438518 5136 flags.go:64] FLAG: --event-qps="50" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438521 5136 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438524 5136 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438527 5136 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438532 5136 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438536 5136 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438539 5136 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438542 5136 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438547 5136 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438552 5136 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438556 5136 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438561 5136 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:48:36.449832 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438565 5136 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438569 5136 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438574 5136 flags.go:64] FLAG: --feature-gates="" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438580 5136 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438584 5136 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438590 5136 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438594 5136 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438599 5136 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438602 5136 flags.go:64] FLAG: --help="false" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438605 5136 flags.go:64] FLAG: --hostname-override="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438615 5136 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438618 5136 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438621 5136 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438625 5136 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438628 5136 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438633 5136 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438637 5136 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438644 5136 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438651 5136 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438656 5136 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438661 5136 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438666 5136 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438670 5136 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438673 5136 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:48:36.450956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438676 5136 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438679 5136 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438683 5136 flags.go:64] FLAG: --lock-file="" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438686 5136 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438689 5136 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438694 5136 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438705 5136 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438711 5136 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438715 5136 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438719 5136 flags.go:64] FLAG: --logging-format="text" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438722 5136 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438726 5136 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438741 5136 flags.go:64] FLAG: --manifest-url="" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438744 5136 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438749 5136 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438752 5136 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438756 5136 flags.go:64] FLAG: --max-pods="110" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438760 5136 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438765 5136 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438771 5136 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438777 5136 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438783 5136 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438791 5136 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438795 5136 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438803 5136 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:48:36.451961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438806 5136 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438811 5136 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438817 5136 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438823 5136 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438828 5136 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438833 5136 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438838 5136 flags.go:64] FLAG: --port="10250" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438842 5136 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438847 5136 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d1f8f1d19832ac46" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438853 5136 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438857 5136 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438862 5136 flags.go:64] FLAG: --register-node="true" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438867 5136 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438874 5136 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438879 5136 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438884 5136 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438890 5136 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438898 5136 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438902 5136 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438906 5136 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438913 5136 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438917 5136 flags.go:64] FLAG: --runonce="false" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438921 5136 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438927 5136 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438932 5136 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438936 5136 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438940 5136 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:48:36.453018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438944 5136 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438947 5136 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438955 5136 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438960 5136 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438964 5136 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438970 5136 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438975 5136 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438981 5136 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438986 5136 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.438992 5136 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439001 5136 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439006 5136 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439010 5136 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439017 5136 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439022 5136 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439027 5136 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439032 5136 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439037 5136 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439042 5136 flags.go:64] FLAG: --v="2" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439049 5136 flags.go:64] FLAG: --version="false" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439054 5136 flags.go:64] FLAG: --vmodule="" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439058 5136 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439063 5136 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439240 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.453947 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439246 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439251 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439254 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439257 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439261 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439268 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439275 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439280 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439284 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439290 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439297 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439302 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439307 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439311 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439316 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439320 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439324 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439329 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439334 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439338 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.454715 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439342 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439346 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439351 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439355 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439360 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439365 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439369 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439374 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439378 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439382 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439386 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439390 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439394 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439401 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439406 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439410 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439415 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439421 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439427 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.458907 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439432 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439436 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439440 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439449 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439454 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439458 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439462 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439467 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439472 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439478 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439482 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439488 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439494 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439499 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439503 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439508 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439513 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439518 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439523 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439527 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.459577 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439531 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439535 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439539 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439543 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439547 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439552 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439557 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439562 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439566 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439570 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439575 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439579 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439583 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439587 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439591 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439597 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439608 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439614 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439619 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.460311 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439623 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439628 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439633 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439639 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439643 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439648 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439653 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439658 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439663 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439668 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439674 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439679 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439684 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439689 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439694 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439699 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439704 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439709 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439714 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.460971 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439721 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439741 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439747 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439752 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439757 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439761 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439766 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439770 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.439816 5136 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439871 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439878 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439884 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439889 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439894 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439900 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439905 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.461704 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439908 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439911 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439913 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439916 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439919 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439922 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439924 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439927 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439931 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439935 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439940 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439944 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439950 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439956 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439960 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439965 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439969 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439973 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439976 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439979 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.462297 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439982 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439986 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439990 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.439995 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440001 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440009 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440015 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440020 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440025 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440030 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440034 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440047 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440052 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440056 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440061 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440066 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440071 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440076 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440080 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440084 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.463115 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440088 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440092 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440096 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440100 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440104 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440109 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440113 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440117 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440122 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440128 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440133 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440137 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440141 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440146 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440151 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440155 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440163 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440171 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440177 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.463945 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440181 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440186 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440191 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440195 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440200 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440203 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440208 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440213 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440217 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440221 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440225 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440230 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440234 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440238 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440240 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440243 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440246 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440248 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440251 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440254 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.464634 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440257 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440260 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440264 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440267 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440270 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440272 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440275 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440278 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440281 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440283 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440288 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440291 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440295 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440299 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440305 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440310 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440314 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440318 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440324 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.465366 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.440328 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.446245 5136 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.446267 5136 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.446292 5136 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.446300 5136 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446384 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446392 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446398 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446404 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446409 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446414 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446418 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446423 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446427 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446432 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446436 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446441 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446445 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446450 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446454 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.466100 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446459 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446463 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446468 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446472 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446476 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446482 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446487 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446491 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446495 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446500 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446505 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446509 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446514 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446518 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446523 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446528 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446532 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446537 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446541 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.466897 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446546 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446550 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446554 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446559 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446563 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446567 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446572 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446576 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446580 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446584 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446589 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446593 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446598 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446602 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446606 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446613 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446619 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446624 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446629 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.467572 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446635 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446641 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446645 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446650 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446656 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446660 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446665 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446669 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446673 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446678 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446683 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446687 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446691 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446695 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446699 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446704 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446708 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446712 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446716 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.468325 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446721 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446725 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446788 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446793 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446797 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446801 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446805 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446810 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446814 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446818 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446823 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446827 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446832 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446836 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446841 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446845 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446849 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446853 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446857 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446862 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.469026 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446867 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446871 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446876 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446881 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446885 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446890 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446894 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446899 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446903 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446907 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446912 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446916 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446921 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.446925 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.446967 5136 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447183 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.469814 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447192 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447197 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447201 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447205 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447210 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447214 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447219 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447223 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447227 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447232 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447237 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447241 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447245 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447249 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447254 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447258 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447263 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447267 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447271 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447276 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.470420 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447281 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447285 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447290 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447294 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447299 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447303 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447307 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447311 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447315 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447319 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447325 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447329 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447333 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447338 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447342 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447348 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447354 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447359 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447364 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.471239 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447368 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447372 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447377 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447381 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447385 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447389 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447393 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447398 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447402 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447406 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447410 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447415 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447419 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447423 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447428 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447432 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447436 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447441 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447445 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447449 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.472129 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447453 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447457 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447462 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447466 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447470 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447474 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447478 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447483 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447487 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447491 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447496 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447503 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447509 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447514 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447519 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447523 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447528 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447532 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447537 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.472887 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447541 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447545 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447549 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447554 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447558 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447562 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447566 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447570 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447575 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447580 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447584 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447588 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447592 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447596 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447600 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447605 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447609 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447613 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447618 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.473558 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447622 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447626 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447630 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447635 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447639 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447646 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447650 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447654 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.447692 5136 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447755 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447762 5136 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447767 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447771 5136 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447775 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447778 5136 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447782 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:36.474243 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447786 5136 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447790 5136 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447794 5136 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447797 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447801 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447806 5136 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447811 5136 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447815 5136 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447819 5136 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447822 5136 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447826 5136 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447831 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447835 5136 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447839 5136 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447843 5136 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447847 5136 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447852 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447855 5136 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447860 5136 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447864 5136 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:36.474796 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447868 5136 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447879 5136 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447883 5136 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447887 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447891 5136 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447896 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447900 5136 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447904 5136 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447909 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447913 5136 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447917 5136 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447922 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447926 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447930 5136 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447934 5136 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447938 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447944 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447948 5136 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447952 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447957 5136 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:36.475474 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447961 5136 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447967 5136 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447972 5136 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447976 5136 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447980 5136 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447984 5136 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447988 5136 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447992 5136 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.447996 5136 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448001 5136 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448005 5136 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448009 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448014 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448020 5136 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448024 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448028 5136 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448033 5136 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448044 5136 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448049 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:36.475990 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448053 5136 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448057 5136 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448061 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448066 5136 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448070 5136 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448074 5136 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448079 5136 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448083 5136 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448087 5136 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448091 5136 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448095 5136 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448099 5136 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448103 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448109 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448113 5136 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448117 5136 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448122 5136 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448128 5136 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448134 5136 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:36.476534 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448138 5136 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448144 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448153 5136 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448157 5136 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448162 5136 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448166 5136 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448170 5136 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448176 5136 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448180 5136 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448185 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448189 5136 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448194 5136 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448198 5136 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448202 5136 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448207 5136 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448211 5136 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448215 5136 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448219 5136 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448224 5136 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:36.477180 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448228 5136 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:36.448232 5136 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.448317 5136 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.449792 5136 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.449856 5136 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.450109 5136 server.go:1014] "Starting client certificate rotation" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.450785 5136 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.452499 5136 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.452539 5136 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:24 +0000 UTC" deadline="2028-05-19 22:17:47.32673299 +0000 UTC" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.452583 5136 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13538h29m10.87415334s" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.452635 5136 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.458444 5136 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.460493 5136 log.go:25] "Validated CRI v1 image API" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.461577 5136 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:48:36.477636 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.463868 5136 fs.go:135] Filesystem UUIDs: map[5114f8f1-5af3-4b4e-8f44-815d5bba4884:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 861710a1-f2b9-4d0d-96f3-faa49fb40a95:/dev/nvme0n1p4] Nov 02 19:48:36.478116 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.463897 5136 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~projected/kube-api-access-lspt9:{mountpoint:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~projected/kube-api-access-lspt9 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~projected/kube-api-access-qlv64:{mountpoint:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~projected/kube-api-access-qlv64 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4588a7b1-e4b1-4255-bf39-d51481fb034f/volumes/kubernetes.io~projected/kube-api-access-sd22g:{mountpoint:/var/lib/kubelet/pods/4588a7b1-e4b1-4255-bf39-d51481fb034f/volumes/kubernetes.io~projected/kube-api-access-sd22g major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~projected/kube-api-access-txk7x:{mountpoint:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~projected/kube-api-access-txk7x major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51d12f6a-8088-4c74-b3d5-7a8867a13872/volumes/kubernetes.io~projected/kube-api-access-x8mvd:{mountpoint:/var/lib/kubelet/pods/51d12f6a-8088-4c74-b3d5-7a8867a13872/volumes/kubernetes.io~projected/kube-api-access-x8mvd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4/volumes/kubernetes.io~projected/kube-api-access-5ffkl:{mountpoint:/var/lib/kubelet/pods/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4/volumes/kubernetes.io~projected/kube-api-access-5ffkl major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62352636-2385-40c9-8beb-6889208afd98/volumes/kubernetes.io~projected/kube-api-access-l755k:{mountpoint:/var/lib/kubelet/pods/62352636-2385-40c9-8beb-6889208afd98/volumes/kubernetes.io~projected/kube-api-access-l755k major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~projected/kube-api-access-kjfkj:{mountpoint:/var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~projected/kube-api-access-kjfkj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~secret/metrics-certs major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~projected/kube-api-access-fm6gt:{mountpoint:/var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~projected/kube-api-access-fm6gt major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~projected/kube-api-access-k46tt:{mountpoint:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~projected/kube-api-access-k46tt major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~projected/kube-api-access-gtvbj:{mountpoint:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~projected/kube-api-access-gtvbj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~projected/kube-api-access-pvr2x:{mountpoint:/var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~projected/kube-api-access-pvr2x major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6842654-0249-4e3b-ade8-d0e46997bed4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b6842654-0249-4e3b-ade8-d0e46997bed4/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~projected/kube-api-access-djtp2:{mountpoint:/var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~projected/kube-api-access-djtp2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e904a0d4-e8e9-4279-bba7-212e6b10fb5d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e904a0d4-e8e9-4279-bba7-212e6b10fb5d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ede0288a-b302-459e-9eaf-8742d594edf3/volumes/kubernetes.io~projected/kube-api-access-v9glh:{mountpoint:/var/lib/kubelet/pods/ede0288a-b302-459e-9eaf-8742d594edf3/volumes/kubernetes.io~projected/kube-api-access-v9glh major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faf3727f-40a7-4555-8aed-2cad99a3cc4d/volumes/kubernetes.io~projected/kube-api-access-zrx5m:{mountpoint:/var/lib/kubelet/pods/faf3727f-40a7-4555-8aed-2cad99a3cc4d/volumes/kubernetes.io~projected/kube-api-access-zrx5m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff2e53d6-22a9-4364-a668-d09a984ea64f/volumes/kubernetes.io~projected/kube-api-access-b5ljq:{mountpoint:/var/lib/kubelet/pods/ff2e53d6-22a9-4364-a668-d09a984ea64f/volumes/kubernetes.io~projected/kube-api-access-b5ljq major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f1502a1e6c2277cf632e8611bd5cfa55208235476b36d9403e2b3968a5a1dfd9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5ebec5dc3210ac9532e9ab1fe382dbe992231c7e76d977919ffe2b67f1ccc73f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8b34ac29ed9b9d9992a406d882bc45f158a3ea06cfcb9e8f177f86afb4e6c97f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3bd967e8f14d43180aa2045d9f1711457ff589414bc7fab8db6f99aefb13ec86/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/7aea327d938f11fb3090ad4e8a2dfa87d7659b6af2e2307c5d63562b401125ee/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/920a6e1506205420f0e6f8c349097d7c42049dc72223ac3b4ce52702e19f938e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3be9b4bbef129d4e7d509a5779f06e1aca6c4db00b6352753962d0a731b51a38/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/570c32bb0783c0a61e32b68fd02ce6b5ccbca7a40e5c1f1823d4383b20a9c83b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/7c894ff0a6ae340bb698ee7e8d402285fc7526ad3541f758938ccb4ab0966426/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a0b596074bebfb3cbce4ac1a4ff0ddd01ec0139c5ff3b98be39191fc50621be6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/97eabde402416d35b94be40bbcbc8334d065d8e9493e58fae6c79e5bbefc60f7/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c012bdd96ba1cc0c03f31a55647de0fea4b238b4277454dab3d48e7c3f375bc4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0811773ac65cc96e9990558210aa62aa70e5b5632b602a3afb64f102eb048c00/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/518fb6a753e0d70e59e9c7ce2e00298470120abebf18ba2f1715df38a252a588/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dd52c7e77bc1a862113c5ea658501d6b920d4816f7b35b447a4f49778e4aabff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ba4d68da1ad99d5dfd5e64511e9d5030eacf74092e87954a341347661f62d3aa/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cc7617d31f7e95b995d31689fd28fa3017967ceafdf6de5b6624419d4d114958/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9c0945d92f164e1ba5fefb90e4300e01c8068784bdbffc15eb334d9cc34c5960/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/16d8a606c54118d6c617feea138870e57da5064e6ae568361387af8d2cf9bee0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5d0061016ec3f45818a36e2f5859b22d6c9a56e1c37cdc9eb6f271f76024feed/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/361155b228c0b9a621e9c4ea4d61929e250eb4ceb6e0e5296a6a575b17c8ea99/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ebece613fc1af2efb288b4d5711db36e924259a57bbec172e5f7eac6962802bf/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/71c78f5430f7c26118ab7988b2326da4b99ddf5bb4ae281cf8e7de36244da125/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5cc13c6cfe9582aa1670886aaa4d742cde1abef4b74f4579142c25f2c0422a6c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/77b15578ffbfa66e90519a5ad1e6a02025d325e62357444db4951d2547863b5f/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/295e710befffae127893ccfb2aa34a42d53f97a28feb5d117ca63d3ff442d9dd/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d4198480d76e8eb6a3d5edb47776d6374e1f39c419999b05322862875981bc5b/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9fec83b36db6e35e8aa907b968fa2864b99118b0235afd7bde90079856d4aebc/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/5a4f15754fcaf48e050eb71204ca08b56ca73b21431ef9f554d4db86a9bd18b4/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/76c65e2bf8271e6608b5b353128e1e8410d8d0e0e9cdad7106375512aecdac8a/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/df4ea70ddc7d8bebbbd037d057fb1c923f66a0198dd828c035424c2a817149fb/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/2f6be05591a753262c230dd66c32c3f24a8883dfa76a1111e701d90e40cd3e36/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/df0d45a13e2c485681ecff06d85aa95be0996c1b455f9c29ee7d1cbbf123043f/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/aad8d983be227f4834b1edd7b2f83f1de7ba576cb6daa38e33a5fbaac1a4cf35/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/bb81dafd5b075b2839a2558dda979619f21c87c2ff6f7cccc662428d8c7a48d7/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/8a407998d4c211e340157dfb1788bf8e74c670265872f875ce36c2d550c8b1b6/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e865f88a337e23f53a2d626050bc824b9172c1e686a1ef390c14db3e33d0d819/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/2a001d7d4063250f25701988f965de636883aa7b0f20cc0e743fa29f7bb40398/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/3092f563d82224f7ce15a1659053f9446290d5a50779e6b09a1a7f3b5f49a234/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/3ff0eae3879b5f26e2d78957ebf3ae9abacc8d48de469d25970bd8981c15ce28/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d1c0bddfe350d30363a6bfac810279d38875a4c6e59434a33554dbd11f2429b4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/9523619ce6b0e9a6f7a87ae6627fc222b9b39022f0cf13d7a43b2eda2b7ac16e/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/10dd823891d3a225a93e7f5aefa9749ad4baf763eeec055d24b475b533ea2647/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/d43f607f357f5ae2a2986ffdd0ab1cd1265a9e5b2a99727aeb8b824d775dcbfd/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/785b3b08da71e35dcbcbf79a7c0910bb48e8d8ad035b8ea1ffb4c056b564a75f/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/9577de4e2aef83966f47ad62ead88726e8d82cd847d8f6f4c94be1ee83882602/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/04028dfcbe5e6ced4d5bf57bae09c3e6773a5bacddc72d29e86d673ade7b2dbc/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/09ec843b5459576c15a4c837a7c5ce75fb2881638c0daf62768c2c3f599248a2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/36e6c98db45f499533b9a9f383284b59f5507f5e07cbac6ccad6a6e5c44c6635/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5bef0223f833d17342f48e2648f9fddec6d27db25d1582a18a89507430f741aa/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/502c855e5256311eb9d9dc08ab16ad00dcc89362de2be9e52fcdc4ae8297fd04/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/57db9d21cfbf970822f9b83989e7fb82f2d0ae3ac8fef8e2ca9076d6b877f4fb/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/23093f513728f7bb258f83563af4957120cb35c0d2718a07c13a8cdd9e21a301/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e1e5fea2bc8c0621ae70b5382b4ff181fd343f3927a46024991da0c0bae15b50/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ca8ce7459f21a96782fdfde69ada45b1a2ecb7e13577f53920e1464f98b61d63/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ef89d8559873dc29164e8f6f3b1f2d854af29fefa0166b71cd1bbda1a2c9337f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/763008385caab8ae546b41c2b8e1f23e27d70973f10820eb45cac3275644ece4/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.473218 5136 manager.go:217] Machine: {Timestamp:2026-11-02 19:48:36.47256343 +0000 UTC m=+0.079107923 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099851 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b243f0c4127f0be85a12534bfacd SystemUUID:ec23b243-f0c4-127f-0be8-5a12534bfacd BootID:ff49ec97-3a5c-4eb9-b5e2-59d752eb52c0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5915351124dc86d5d4c5820f876bc92e88c0cfd9d816b6e2a94b5770f3e3da47/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e904a0d4-e8e9-4279-bba7-212e6b10fb5d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~projected/kube-api-access-kjfkj DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0132096f41ded9d772f67173e3bed8ee340351958a7a41df15d7d8b58ae98afd/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f10550abd6ed217ba02ed841e380e9f15ae2be322a8220ca6977e72cc3d409f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8741ab68e84e4e9a73e6fc4ce43b4c7481456716b048b384b0dd2a37155291bf/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~projected/kube-api-access-djtp2 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~projected/kube-api-access-gtvbj DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74f52684ef4761fae5d1eb83980816cc6aea030e75c638a4b633d08e31f984eb/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~projected/kube-api-access-txk7x DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f27128809705af9e7aed63f0378ca6098315c695c50e4730cbb28dcd796a936/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fa330d2a283321230de2f0508a85d5bce4679550ce7930293f51a4383be8a68/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~projected/kube-api-access-fm6gt DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8e62c079776c517fd1df99e90c998acbf1652c9915d091bc945094c5dba097e/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~projected/kube-api-access-lspt9 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4588a7b1-e4b1-4255-bf39-d51481fb034f/volumes/kubernetes.io~projected/kube-api-access-sd22g DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d10ea0b79c44dc8196e020cd21f760edbdae972ac17ffa144e3d23c3caf54260/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66d713cc-a097-4c92-bafe-e899ece7bc54/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ab033f0fb062cef0d7169985adee7a9151484cb95976aacaa6f9f1341f2d525/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6842654-0249-4e3b-ade8-d0e46997bed4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67f7a8897d8e3d168f9694fcf44913be02aad226db83aeca2ff263842cf21949/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad4cb29-a4bf-4374-8422-f9b42da120fb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55fbd6777be4cd5ee6f1a85edbe8473dd25d43926daaeee2c431bf092823ac14/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04384d3df782d1c7b7af552bba8e790fb9b0055a9827e380dd4ca4bfc52ef0b5/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c/volumes/kubernetes.io~projected/kube-api-access-pvr2x DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~projected/kube-api-access-k46tt DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f7be2ab44505eb4ff2410b7527b894cd86b8cd7ad4943681506472d684498aa/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff2e53d6-22a9-4364-a668-d09a984ea64f/volumes/kubernetes.io~projected/kube-api-access-b5ljq DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~projected/kube-api-access-qlv64 DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51d12f6a-8088-4c74-b3d5-7a8867a13872/volumes/kubernetes.io~projected/kube-api-access-x8mvd DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4/volumes/kubernetes.io~projected/kube-api-access-5ffkl DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e198c8b21df99e737f014241d8584e1e52fb203b825ba035c083be3bbdcb4b4c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ede0288a-b302-459e-9eaf-8742d594edf3/volumes/kubernetes.io~projected/kube-api-access-v9glh DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/faf3727f-40a7-4555-8aed-2cad99a3cc4d/volumes/kubernetes.io~projected/kube-api-access-zrx5m DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccc35fe2706798b9f863e18da93257fb2671ca55fbe3eb9df8847b0f78328224/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc3f4fe08d9fca8f5441a6fae18c6148aeceb9e2b2a0afcb3f0768f551ee0aaf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/62352636-2385-40c9-8beb-6889208afd98/volumes/kubernetes.io~projected/kube-api-access-l755k DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bfa771b-a624-46be-8b7a-869cc29cb8c4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0132096f41ded9d MacAddress:86:a5:17:90:6a:c0 Speed:10000 Mtu:8901} {Name:5915351124dc86d MacAddress:62:7a:8e:f1:29:af Speed:10000 Mtu:8901} {Name:67f7a8897d8e3d1 MacAddress:72:db:b0:f9:86:15 Speed:10000 Mtu:8901} {Name:74f52684ef4761f MacAddress:8a:84:e4:c5:79:4c Speed:10000 Mtu:8901} {Name:7ab033f0fb062ce MacAddress:fe:85:28:b9:96:20 Speed:10000 Mtu:8901} {Name:9f27128809705af MacAddress:b2:85:e8:d2:94:d0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f2:1b:79:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:3d:45:ad:88:89 Speed:0 Mtu:8901} {Name:c8e62c079776c51 MacAddress:16:16:f4:63:7d:f5 Speed:10000 Mtu:8901} {Name:d10ea0b79c44dc8 MacAddress:1a:69:54:fe:73:10 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f2:1b:79:51 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:49:a6:69:32:3e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:62:cb:99:ea:f4:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.473663 5136 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.473720 5136 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474061 5136 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474084 5136 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-34.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474271 5136 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474282 5136 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474297 5136 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474328 5136 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474410 5136 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474664 5136 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474825 5136 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474878 5136 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474899 5136 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474911 5136 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474922 5136 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.474933 5136 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.476516 5136 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.476528 5136 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:48:36.478353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.477027 5136 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:48:36.489562 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.489542 5136 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:48:36.489562 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.489564 5136 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:48:36.489769 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.489704 5136 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:48:36.490042 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490015 5136 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:48:36.490098 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490063 5136 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:48:36.490259 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490225 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490273 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490286 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490295 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490304 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490313 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:48:36.490369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490351 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:48:36.490607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490547 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:48:36.490607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490567 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:48:36.490607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490582 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:48:36.490607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490596 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:48:36.490800 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490614 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:48:36.490800 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490666 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:48:36.490800 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.490677 5136 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:48:36.495684 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.495137 5136 server.go:1263] "Started kubelet" Nov 02 19:48:36.495684 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.495195 5136 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:48:36.495684 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.495331 5136 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:48:36.495684 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.495397 5136 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:48:36.495922 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.495704 5136 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:48:36.495795 ip-10-0-128-34 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:48:36.498073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.496308 5136 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:48:36.502080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502062 5136 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502172 5136 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.502223 5136 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502215 5136 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:25 +0000 UTC" deadline="2028-05-31 03:47:28.585159009 +0000 UTC" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502248 5136 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13807h58m52.082914306s" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502270 5136 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502279 5136 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:48:36.502811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.502368 5136 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:48:36.503976 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.503949 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62352636-2385-40c9-8beb-6889208afd98" volumeName="kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k" seLinuxMountContext="" Nov 02 19:48:36.503976 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.503975 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c" volumeName="kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.503984 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6842654-0249-4e3b-ade8-d0e46997bed4" volumeName="kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.503992 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e904a0d4-e8e9-4279-bba7-212e6b10fb5d" volumeName="kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.503998 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf3727f-40a7-4555-8aed-2cad99a3cc4d" volumeName="kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504006 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d90c8b2-351c-45f5-a22c-84a64a1a9fd4" volumeName="kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504013 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4588a7b1-e4b1-4255-bf39-d51481fb034f" volumeName="kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504020 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d12f6a-8088-4c74-b3d5-7a8867a13872" volumeName="kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504028 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4" volumeName="kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504035 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66d713cc-a097-4c92-bafe-e899ece7bc54" volumeName="kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504059 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504068 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c" volumeName="kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504083 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d64732a9-9d3a-4d5e-8b01-f5ebef4a986c" volumeName="kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504090 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad4cb29-a4bf-4374-8422-f9b42da120fb" volumeName="kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned" seLinuxMountContext="" Nov 02 19:48:36.504095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504099 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad4cb29-a4bf-4374-8422-f9b42da120fb" volumeName="kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504106 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4588a7b1-e4b1-4255-bf39-d51481fb034f" volumeName="kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504113 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504120 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ba6228-6606-4b2b-a7b0-04f13a9e7e0c" volumeName="kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504128 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ba6228-6606-4b2b-a7b0-04f13a9e7e0c" volumeName="kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504135 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d64732a9-9d3a-4d5e-8b01-f5ebef4a986c" volumeName="kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504143 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf3727f-40a7-4555-8aed-2cad99a3cc4d" volumeName="kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504150 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad4cb29-a4bf-4374-8422-f9b42da120fb" volumeName="kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504157 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d90c8b2-351c-45f5-a22c-84a64a1a9fd4" volumeName="kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504163 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504171 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d12f6a-8088-4c74-b3d5-7a8867a13872" volumeName="kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504178 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c" volumeName="kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504185 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504195 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ba6228-6606-4b2b-a7b0-04f13a9e7e0c" volumeName="kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504206 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6842654-0249-4e3b-ade8-d0e46997bed4" volumeName="kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504212 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d64732a9-9d3a-4d5e-8b01-f5ebef4a986c" volumeName="kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504219 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d64732a9-9d3a-4d5e-8b01-f5ebef4a986c" volumeName="kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504226 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf3727f-40a7-4555-8aed-2cad99a3cc4d" volumeName="kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504232 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff2e53d6-22a9-4364-a668-d09a984ea64f" volumeName="kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504239 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d90c8b2-351c-45f5-a22c-84a64a1a9fd4" volumeName="kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504246 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504252 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504260 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504266 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504273 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c" volumeName="kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504282 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" volumeName="kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504292 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d90c8b2-351c-45f5-a22c-84a64a1a9fd4" volumeName="kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504302 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d12f6a-8088-4c74-b3d5-7a8867a13872" volumeName="kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504314 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bfa771b-a624-46be-8b7a-869cc29cb8c4" volumeName="kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504325 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ba6228-6606-4b2b-a7b0-04f13a9e7e0c" volumeName="kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504338 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf3727f-40a7-4555-8aed-2cad99a3cc4d" volumeName="kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504350 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bfa771b-a624-46be-8b7a-869cc29cb8c4" volumeName="kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504363 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504374 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504389 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4" volumeName="kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504403 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66d713cc-a097-4c92-bafe-e899ece7bc54" volumeName="kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504415 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ba6228-6606-4b2b-a7b0-04f13a9e7e0c" volumeName="kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504427 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede0288a-b302-459e-9eaf-8742d594edf3" volumeName="kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh" seLinuxMountContext="" Nov 02 19:48:36.504429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504440 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff2e53d6-22a9-4364-a668-d09a984ea64f" volumeName="kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq" seLinuxMountContext="" Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504454 5136 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb1ec8-149f-4d4b-a5b6-728048a4a0e2" volumeName="kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504467 5136 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.504474 5136 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505204 5136 factory.go:55] Registering systemd factory Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505219 5136 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505548 5136 factory.go:153] Registering CRI-O factory Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505560 5136 factory.go:223] Registration of the crio container factory successfully Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505607 5136 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505623 5136 factory.go:103] Registering Raw factory Nov 02 19:48:36.505865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505637 5136 manager.go:1199] Started watching for new ooms in manager Nov 02 19:48:36.506295 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.505994 5136 manager.go:319] Starting recovery of all containers Nov 02 19:48:36.537008 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.536961 5136 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:48:36.538536 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.538512 5136 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:48:36.538536 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.538535 5136 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:48:36.538720 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.538562 5136 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:48:36.538720 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.538608 5136 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:48:36.566150 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.566112 5136 generic.go:356] "Generic (PLEG): container finished" podID="6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2" containerID="7cd8e5ae68cf64b095485ef43062ac3f484c406c5f4e114f6d53ad9a7c13d318" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581581 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="3c35d5dd913040ca819262384f7f7c3763d9f361db8e6dfef98c1dda81c0fe38" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581604 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="249191cb96d933209ffe62b75633e6a66f0a776f0d75199003c9049589a5570a" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581612 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="35ae59d244970345dcb6f7c1e1570777855304825987387d54f0fa629c83b406" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581618 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="cd8235ecc9ff79e1e71a2c5c1c35ea1603fbef0270968a25f62820d99d3b9a1d" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581623 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="3a949bee168830b6e42394e8a48f634e1294a9b6afa80eb3c0895e6666f0838e" exitCode=0 Nov 02 19:48:36.581643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.581629 5136 generic.go:356] "Generic (PLEG): container finished" podID="faf3727f-40a7-4555-8aed-2cad99a3cc4d" containerID="e1838825d5d683ea6cac2c0deeb657fb26bdefe5b946013b385a0bb340aded1d" exitCode=0 Nov 02 19:48:36.591405 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.591380 5136 generic.go:356] "Generic (PLEG): container finished" podID="fbc6ff8a97420718ecc4211b6b597a48" containerID="48d6f39bc9840d5f2c2916ba00ce215e2c7df27c47a54e7bcae33b9e9660408c" exitCode=0 Nov 02 19:48:36.598599 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.598581 5136 manager.go:324] Recovery completed Nov 02 19:48:36.602363 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.602345 5136 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:48:36.613833 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.613817 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.615792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.615777 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.615859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.615805 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.615859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.615820 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.618776 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.618754 5136 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:48:36.618776 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.618776 5136 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:48:36.618935 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.618797 5136 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:48:36.618998 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.618980 5136 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:48:36.619065 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619000 5136 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:48:36.619065 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619025 5136 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:48:36.619065 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619033 5136 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:48:36.619065 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619040 5136 policy_none.go:50] "Start" Nov 02 19:48:36.619065 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619058 5136 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:48:36.619266 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619070 5136 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:48:36.619309 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619277 5136 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:48:36.619309 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619287 5136 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:48:36.619309 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619295 5136 policy_none.go:44] "Start" Nov 02 19:48:36.619797 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.619778 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.621100 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.621082 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.621185 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.621112 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.621185 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.621123 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.623930 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.623904 5136 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:48:36.627347 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627334 5136 manager.go:341] "Starting Device Plugin manager" Nov 02 19:48:36.627425 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.627358 5136 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:48:36.627425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627368 5136 server.go:87] "Starting device plugin registration server" Nov 02 19:48:36.627633 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627619 5136 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:48:36.627691 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627632 5136 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:48:36.627762 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627699 5136 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:48:36.627859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627818 5136 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:48:36.627966 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627863 5136 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:48:36.627966 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627870 5136 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:48:36.628066 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.627938 5136 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:48:36.627853639Z","UUID":"b7b94355-56f4-4e4b-b6ef-84920413c24c","Handler":null,"Name":"","Endpoint":""} Nov 02 19:48:36.629431 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.629403 5136 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:48:36.629505 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.629440 5136 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" Nov 02 19:48:36.629825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.629748 5136 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:48:36.629825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.629784 5136 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:48:36.638692 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.638655 5136 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal"] Nov 02 19:48:36.638904 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.638888 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.639913 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.639897 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.639991 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.639927 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.639991 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.639941 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.640235 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.640220 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.640359 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.640345 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.641213 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641199 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.641287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641227 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.641287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641241 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.641287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641198 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.641482 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641299 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.641482 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641309 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.641482 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.641421 5136 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.641626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641536 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"d53a41f215f25804a365e963f8c63305","Type":"ContainerStarted","Data":"04f53fbad0faecfff8b96e7b20d8d323689495685fa20d2ccd7e55e0e5d1d2d2"} Nov 02 19:48:36.641626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641585 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"d53a41f215f25804a365e963f8c63305","Type":"ContainerStarted","Data":"e5d008272e0ad6b72e14c1e66ff16e670e4826bf0a63451ce9d468f2e8a1dcd8"} Nov 02 19:48:36.641722 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641703 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.641722 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641709 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerStarted","Data":"4c71b35a86eb4809d0999a4b2af083adf53fe0cea0fb9afb141c0c4fabf481fb"} Nov 02 19:48:36.641820 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641748 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerDied","Data":"48d6f39bc9840d5f2c2916ba00ce215e2c7df27c47a54e7bcae33b9e9660408c"} Nov 02 19:48:36.641820 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.641770 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" event={"ID":"fbc6ff8a97420718ecc4211b6b597a48","Type":"ContainerStarted","Data":"80870fc3553a4a40f59fe63a1ca9edf7950a92744b9adc7a3d10a0fdaca5a2b8"} Nov 02 19:48:36.642591 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.642572 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.642666 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.642605 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.642666 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.642619 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.642813 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.642778 5136 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.us-west-2.compute.internal\" not found" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.674840 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.674817 5136 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:36.674917 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.674828 5136 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:36.674917 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.674899 5136 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:48:36.675885 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.675870 5136 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:36.703302 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.703281 5136 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.728067 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.728051 5136 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:36.729086 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.729073 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:36.729156 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.729099 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:36.729156 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.729109 5136 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:36.729156 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.729143 5136 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.769257 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.769240 5136 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:36.769497 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.769483 5136 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:36.769543 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.769518 5136 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.769543 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.769530 5136 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.775551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.775535 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.775606 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.775559 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.775606 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.775577 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.823669 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.823644 5136 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:36.823800 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:36.823686 5136 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.831548 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.831530 5136 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.831613 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.831594 5136 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876253 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876281 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876297 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876338 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876374 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d53a41f215f25804a365e963f8c63305-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"d53a41f215f25804a365e963f8c63305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:36.876521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:36.876374 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc6ff8a97420718ecc4211b6b597a48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" (UID: \"fbc6ff8a97420718ecc4211b6b597a48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:37.476166 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.476137 5136 apiserver.go:52] "Watching apiserver" Nov 02 19:48:37.574338 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.574309 5136 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:48:37.574829 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.574805 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-2jzk4","openshift-image-registry/node-ca-5s9lr","kube-system/konnectivity-agent-wfdgc","openshift-dns/dns-default-nxh58","openshift-ovn-kubernetes/ovnkube-node-bff7p","openshift-network-diagnostics/network-check-target-nvcwl","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h","openshift-cluster-node-tuning-operator/tuned-hsvmf","openshift-multus/multus-additional-cni-plugins-xwh2b","openshift-monitoring/node-exporter-xb77n","kube-system/global-pull-secret-syncer-h6fsm","openshift-dns/node-resolver-zpmbp","openshift-ingress-canary/ingress-canary-jwdhv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-tf5w7","openshift-network-operator/iptables-alerter-9hdxl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl","openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc","kube-system/kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal","openshift-insights/insights-runtime-extractor-hspfn"] Nov 02 19:48:37.594267 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.594245 5136 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:37.604597 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.604579 5136 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:48:37.626945 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.626919 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:48:37.627046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.626999 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:48:37.627046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.627018 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.627046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.627024 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.627322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.627307 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:48:37.627855 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.627832 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:48:37.627939 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.627900 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\"" Nov 02 19:48:37.628297 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.628274 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.628363 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.628311 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.628993 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.628974 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\"" Nov 02 19:48:37.629077 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629013 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\"" Nov 02 19:48:37.629182 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629160 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.629234 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629218 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:48:37.629394 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629378 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:48:37.629497 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629170 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\"" Nov 02 19:48:37.629569 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629550 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.629655 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629639 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\"" Nov 02 19:48:37.629775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629761 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.629896 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629763 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.629956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629924 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:48:37.629956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629932 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\"" Nov 02 19:48:37.630047 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.629986 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:48:37.630157 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630141 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\"" Nov 02 19:48:37.630362 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630344 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.630425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630374 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:48:37.630845 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630828 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.630927 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630890 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:48:37.630927 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630902 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:48:37.631017 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.630976 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:48:37.631335 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631073 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.631335 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631245 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:48:37.631514 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631493 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:48:37.631699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631634 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.631699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631647 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.631699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631666 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:48:37.631964 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631837 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\"" Nov 02 19:48:37.631964 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.631941 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\"" Nov 02 19:48:37.632105 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632088 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.632154 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632127 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.632199 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632160 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.632465 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632449 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:48:37.632590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632575 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:48:37.632776 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.632761 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:48:37.678711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.678688 5136 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:37.678826 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:37.678752 5136 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.us-west-2.compute.internal" Nov 02 19:48:37.680901 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680877 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680911 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.681006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680928 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.681006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680941 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.681006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680959 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.681006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.680985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681081 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681136 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.681217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681174 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.681217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681200 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681402 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681246 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681402 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681257 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4588a7b1-e4b1-4255-bf39-d51481fb034f-iptables-alerter-script\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.681402 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681313 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfkj\" (UniqueName: \"kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:37.681528 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681415 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.681528 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681430 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.681528 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681435 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.681528 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681481 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-cni-binary-copy\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.681676 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681555 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.681676 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681592 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.681676 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681636 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.681676 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681670 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.681828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681752 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-tmp-dir\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.681828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681756 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.681828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681811 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.681929 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681858 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.681959 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681928 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.681995 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.681959 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.682035 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682015 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-script-lib\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682042 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.682106 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682071 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.682106 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682099 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.682168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682106 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.682168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682124 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682156 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtvbj\" (UniqueName: \"kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682266 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682181 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.682266 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682186 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.682266 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682228 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9glh\" (UniqueName: \"kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.682401 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682265 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.682401 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682309 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-original-pull-secret\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.682401 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682324 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.682401 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682357 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.682401 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682388 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.682558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682424 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682452 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.682558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682495 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682549 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.682675 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682586 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.682675 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682629 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.682675 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682663 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovnkube-config\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.682798 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682690 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:48:37.682798 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682725 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:37.682864 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682808 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.682864 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682841 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.682864 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682842 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6842654-0249-4e3b-ade8-d0e46997bed4-konnectivity-ca\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:48:37.682955 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682917 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkl\" (UniqueName: \"kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.682990 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682954 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.682990 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.682980 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspt9\" (UniqueName: \"kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.683062 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683006 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.683062 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683036 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.683149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683060 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mvd\" (UniqueName: \"kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.683149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683089 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:48:37.683149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683117 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.683285 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683145 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.683285 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683235 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.683285 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683276 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683299 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683321 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-tmp\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683323 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683364 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrx5m\" (UniqueName: \"kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683384 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6842654-0249-4e3b-ade8-d0e46997bed4-agent-certs\") pod \"konnectivity-agent-wfdgc\" (UID: \"b6842654-0249-4e3b-ade8-d0e46997bed4\") " pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:48:37.683399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683385 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.683590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683442 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.683590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683477 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd22g\" (UniqueName: \"kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.683590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683515 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.683590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683539 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.683590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683558 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46tt\" (UniqueName: \"kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.683764 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683604 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.683764 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683682 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.683764 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683712 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.683861 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683776 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.683861 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683833 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.683955 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683898 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-tmp-dir\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.683955 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683916 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.683955 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683940 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.684063 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.683977 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.684107 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684051 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.684107 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684103 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684193 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684142 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.684193 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684189 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684265 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684214 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684265 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684231 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684265 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684240 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.684365 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684294 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-tuned\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684365 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684297 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:37.684426 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684404 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.684461 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684435 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlv64\" (UniqueName: \"kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.684496 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684469 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff2e53d6-22a9-4364-a668-d09a984ea64f-serviceca\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.684496 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684463 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.684572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684507 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.684572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684517 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66d713cc-a097-4c92-bafe-e899ece7bc54-metrics-certs\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:37.684572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684533 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.684682 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684585 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.684682 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684615 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.684682 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684642 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.684786 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684665 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtp2\" (UniqueName: \"kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.684786 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684713 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.684786 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684759 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.684899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684783 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:37.684899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684810 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.684984 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684912 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.684984 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684937 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-textfile\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.684984 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684946 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.684984 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.684970 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.685094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685003 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.685094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685046 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.685094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685076 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:37.685094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685082 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-data-volume\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.685275 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685104 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.685275 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685193 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.685275 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685213 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.685275 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685232 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685283 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvr2x\" (UniqueName: \"kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685313 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685342 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685390 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685432 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk7x\" (UniqueName: \"kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685456 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-env-overrides\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.685507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685481 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.685784 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685529 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ljq\" (UniqueName: \"kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.685784 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685544 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-daemon-config\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.685784 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685558 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.685784 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685585 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.685889 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.685816 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-ovn-node-metrics-cert\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.729980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.729927 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:48:37.730152 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730135 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:48:37.730200 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730141 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:48:37.730361 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730342 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:48:37.730507 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730492 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.730614 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730600 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.730701 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730688 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dzphr\"" Nov 02 19:48:37.730762 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730697 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730906 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730919 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730935 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730938 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730907 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:48:37.730980 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730958 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730989 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.730997 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731028 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731134 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731148 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k8mpj\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731179 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:48:37.731211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731189 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\"" Nov 02 19:48:37.731418 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731221 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\"" Nov 02 19:48:37.731418 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731231 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:48:37.731418 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731238 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\"" Nov 02 19:48:37.731418 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731262 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:48:37.731418 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.731289 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8hmw\"" Nov 02 19:48:37.732581 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.732563 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.732620 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.732608 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.733751 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733726 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.733825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733784 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.733825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733817 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.733925 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733827 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-metrics-tls\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.733925 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733857 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-config-volume\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.733925 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733896 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.734016 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733927 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.734016 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.733979 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.734673 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.734643 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfkj\" (UniqueName: \"kubernetes.io/projected/66d713cc-a097-4c92-bafe-e899ece7bc54-kube-api-access-kjfkj\") pod \"network-metrics-daemon-tf5w7\" (UID: \"66d713cc-a097-4c92-bafe-e899ece7bc54\") " pod="openshift-multus/network-metrics-daemon-tf5w7" Nov 02 19:48:37.734746 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.734708 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.734864 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.734844 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.734901 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.734848 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.734975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.734962 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtvbj\" (UniqueName: \"kubernetes.io/projected/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-kube-api-access-gtvbj\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.735068 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.735051 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfa771b-a624-46be-8b7a-869cc29cb8c4-cert\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:37.735301 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.735286 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9glh\" (UniqueName: \"kubernetes.io/projected/ede0288a-b302-459e-9eaf-8742d594edf3-kube-api-access-v9glh\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.735971 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.735957 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-metrics-client-ca\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.736010 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.735976 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-tls\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.736759 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.736723 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrx5m\" (UniqueName: \"kubernetes.io/projected/faf3727f-40a7-4555-8aed-2cad99a3cc4d-kube-api-access-zrx5m\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.736802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.736743 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd22g\" (UniqueName: \"kubernetes.io/projected/4588a7b1-e4b1-4255-bf39-d51481fb034f-kube-api-access-sd22g\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.736802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.736778 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mvd\" (UniqueName: \"kubernetes.io/projected/51d12f6a-8088-4c74-b3d5-7a8867a13872-kube-api-access-x8mvd\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.736910 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.736895 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/6bfa771b-a624-46be-8b7a-869cc29cb8c4-kube-api-access-fm6gt\") pod \"ingress-canary-jwdhv\" (UID: \"6bfa771b-a624-46be-8b7a-869cc29cb8c4\") " pod="openshift-ingress-canary/ingress-canary-jwdhv" Nov 02 19:48:37.738277 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738255 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk7x\" (UniqueName: \"kubernetes.io/projected/45fb1ec8-149f-4d4b-a5b6-728048a4a0e2-kube-api-access-txk7x\") pod \"kube-state-metrics-5dc8d7d89b-4rc8h\" (UID: \"45fb1ec8-149f-4d4b-a5b6-728048a4a0e2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4rc8h" Nov 02 19:48:37.738436 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738362 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46tt\" (UniqueName: \"kubernetes.io/projected/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-kube-api-access-k46tt\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.738528 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738447 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtp2\" (UniqueName: \"kubernetes.io/projected/d64732a9-9d3a-4d5e-8b01-f5ebef4a986c-kube-api-access-djtp2\") pod \"dns-default-nxh58\" (UID: \"d64732a9-9d3a-4d5e-8b01-f5ebef4a986c\") " pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:37.738956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738774 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l755k\" (UniqueName: \"kubernetes.io/projected/62352636-2385-40c9-8beb-6889208afd98-kube-api-access-l755k\") pod \"network-check-target-nvcwl\" (UID: \"62352636-2385-40c9-8beb-6889208afd98\") " pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:37.738956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738791 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ljq\" (UniqueName: \"kubernetes.io/projected/ff2e53d6-22a9-4364-a668-d09a984ea64f-kube-api-access-b5ljq\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.738956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738873 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffkl\" (UniqueName: \"kubernetes.io/projected/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-kube-api-access-5ffkl\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.738956 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738946 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspt9\" (UniqueName: \"kubernetes.io/projected/0ad4cb29-a4bf-4374-8422-f9b42da120fb-kube-api-access-lspt9\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.739139 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738971 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlv64\" (UniqueName: \"kubernetes.io/projected/0d90c8b2-351c-45f5-a22c-84a64a1a9fd4-kube-api-access-qlv64\") pod \"openshift-state-metrics-7bd7489957-8dvlc\" (UID: \"0d90c8b2-351c-45f5-a22c-84a64a1a9fd4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dvlc" Nov 02 19:48:37.739139 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.738999 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvr2x\" (UniqueName: \"kubernetes.io/projected/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-kube-api-access-pvr2x\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.786615 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786591 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786625 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786648 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786671 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.786711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786700 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-netns\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786705 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-multus-certs\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786709 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786716 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-bin\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786763 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786780 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4-hosts-file\") pod \"node-resolver-zpmbp\" (UID: \"5ce2cd6d-38ca-4f4f-87fa-0b12f8bef4d4\") " pod="openshift-dns/node-resolver-zpmbp" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786782 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c-crio-socket\") pod \"insights-runtime-extractor-hspfn\" (UID: \"8ac4ccdd-a8e0-484d-acb2-fa7fddc0da3c\") " pod="openshift-insights/insights-runtime-extractor-hspfn" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786784 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786805 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786823 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786840 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-kubelet\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786851 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786857 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-ovn\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786875 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786901 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786910 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff2e53d6-22a9-4364-a668-d09a984ea64f-host\") pod \"node-ca-5s9lr\" (UID: \"ff2e53d6-22a9-4364-a668-d09a984ea64f\") " pod="openshift-image-registry/node-ca-5s9lr" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786924 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786935 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-kubelet\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786941 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-system-cni-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786947 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786960 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-run-systemd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786968 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786977 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-run\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.786990 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787018 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-systemd\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787018 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787045 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787068 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787095 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787109 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-conf\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787122 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-sys-fs\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787143 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-node-exporter-wtmp\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787146 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787161 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-kubelet-config\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787160 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-device-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787173 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787193 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-netns\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787211 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysconfig\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787251 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787279 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787304 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787326 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787305 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-etc-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787351 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787388 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-socket-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787389 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-slash\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787396 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-systemd-units\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787408 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787430 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-lib-modules\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787438 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787466 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787491 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787503 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-bin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787511 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-modprobe-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787537 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-sys\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787591 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787610 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-sysctl-d\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787620 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787643 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787648 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-var-lib-cni-multus\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787673 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787693 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787698 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787700 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e904a0d4-e8e9-4279-bba7-212e6b10fb5d-dbus\") pod \"global-pull-secret-syncer-h6fsm\" (UID: \"e904a0d4-e8e9-4279-bba7-212e6b10fb5d\") " pod="kube-system/global-pull-secret-syncer-h6fsm" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787758 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-socket-dir-parent\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787770 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787794 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-host-run-k8s-cni-cncf-io\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787771 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.787770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787800 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787825 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787849 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787858 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faf3727f-40a7-4555-8aed-2cad99a3cc4d-cnibin\") pod \"multus-additional-cni-plugins-xwh2b\" (UID: \"faf3727f-40a7-4555-8aed-2cad99a3cc4d\") " pod="openshift-multus/multus-additional-cni-plugins-xwh2b" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787875 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787890 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-system-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787899 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787906 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-registration-dir\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787914 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-sys\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787924 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787947 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787967 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-var-lib-openvswitch\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.787971 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788014 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-node-log\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788034 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-var-lib-kubelet\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788045 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788012 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-cni-netd\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788068 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788083 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788097 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788118 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788132 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-log-socket\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788135 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-conf-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788141 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788165 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-cnibin\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788169 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-etc-kubernetes\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788178 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ede0288a-b302-459e-9eaf-8742d594edf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7wzl\" (UID: \"ede0288a-b302-459e-9eaf-8742d594edf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7wzl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788188 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788210 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788226 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788234 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2-root\") pod \"node-exporter-xb77n\" (UID: \"6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2\") " pod="openshift-monitoring/node-exporter-xb77n" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788242 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788260 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788274 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788277 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788291 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4588a7b1-e4b1-4255-bf39-d51481fb034f-host-slash\") pod \"iptables-alerter-9hdxl\" (UID: \"4588a7b1-e4b1-4255-bf39-d51481fb034f\") " pod="openshift-network-operator/iptables-alerter-9hdxl" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788189 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-multus-cni-dir\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788298 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51d12f6a-8088-4c74-b3d5-7a8867a13872-hostroot\") pod \"multus-2jzk4\" (UID: \"51d12f6a-8088-4c74-b3d5-7a8867a13872\") " pod="openshift-multus/multus-2jzk4" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788310 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-host\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788319 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788330 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad4cb29-a4bf-4374-8422-f9b42da120fb-etc-kubernetes\") pod \"tuned-hsvmf\" (UID: \"0ad4cb29-a4bf-4374-8422-f9b42da120fb\") " pod="openshift-cluster-node-tuning-operator/tuned-hsvmf" Nov 02 19:48:37.789322 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.788356 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ba6228-6606-4b2b-a7b0-04f13a9e7e0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bff7p\" (UID: \"81ba6228-6606-4b2b-a7b0-04f13a9e7e0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:37.877836 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:37.877817 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:48:38.593787 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.593751 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h"] Nov 02 19:48:38.596992 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.596959 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.649236 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.649210 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 19:48:38.650088 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.650059 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h"] Nov 02 19:48:38.652293 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.652273 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7bppm\"" Nov 02 19:48:38.695115 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.695091 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d6f4284-710c-441c-973d-d61c90b9443f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.695211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.695152 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mcf\" (UniqueName: \"kubernetes.io/projected/3d6f4284-710c-441c-973d-d61c90b9443f-kube-api-access-g4mcf\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.796111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.796086 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mcf\" (UniqueName: \"kubernetes.io/projected/3d6f4284-710c-441c-973d-d61c90b9443f-kube-api-access-g4mcf\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.796206 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.796132 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d6f4284-710c-441c-973d-d61c90b9443f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.796447 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.796430 5136 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:48:38.798310 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.798283 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d6f4284-710c-441c-973d-d61c90b9443f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.850207 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.850157 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mcf\" (UniqueName: \"kubernetes.io/projected/3d6f4284-710c-441c-973d-d61c90b9443f-kube-api-access-g4mcf\") pod \"monitoring-plugin-77f4f76594-q2q9h\" (UID: \"3d6f4284-710c-441c-973d-d61c90b9443f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:38.906975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:38.906953 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:39.264889 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:39.264860 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h"] Nov 02 19:48:39.268478 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:39.268432 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6f4284_710c_441c_973d_d61c90b9443f.slice/crio-341c219bf0ad7f7943349739c37033409b504fa4ebdc72e3db31eaf573b387a9 WatchSource:0}: Error finding container 341c219bf0ad7f7943349739c37033409b504fa4ebdc72e3db31eaf573b387a9: Status 404 returned error can't find the container with id 341c219bf0ad7f7943349739c37033409b504fa4ebdc72e3db31eaf573b387a9 Nov 02 19:48:39.343211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:39.343191 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nxh58" Nov 02 19:48:39.603647 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:39.603373 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" event={"ID":"3d6f4284-710c-441c-973d-d61c90b9443f","Type":"ContainerStarted","Data":"341c219bf0ad7f7943349739c37033409b504fa4ebdc72e3db31eaf573b387a9"} Nov 02 19:48:39.605492 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:39.605465 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nvcwl" event={"ID":"62352636-2385-40c9-8beb-6889208afd98","Type":"ContainerStarted","Data":"78bb08205ebcb598106b971d804690a74b8f97042aedc818de35231609a6a659"} Nov 02 19:48:41.613453 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:41.613415 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" event={"ID":"3d6f4284-710c-441c-973d-d61c90b9443f","Type":"ContainerStarted","Data":"39267f58ef1baaa1716e10f04d3d0b911d2646289b62ceb6f7832587aecf417e"} Nov 02 19:48:41.722192 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:41.722149 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" podStartSLOduration=1.611651352 podStartE2EDuration="3.722135375s" totalImagesPullingTime="2.110484023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:41.721876817 +0000 UTC m=+5.328421315" watchObservedRunningTime="2026-11-02 19:48:41.722135375 +0000 UTC m=+5.328679870" Nov 02 19:48:42.616645 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:42.616620 5136 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:48:43.218242 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:43.218216 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:44.376979 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.376943 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-995647d55-2ckjz"] Nov 02 19:48:44.379419 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.379398 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.431617 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.431592 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 19:48:44.431760 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.431675 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:48:44.476626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.476590 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 19:48:44.476626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.476596 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k2k8s\"" Nov 02 19:48:44.476828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.476632 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 19:48:44.479152 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.479134 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4mrp37tkte2v3\"" Nov 02 19:48:44.480181 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.480162 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-995647d55-2ckjz"] Nov 02 19:48:44.526581 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526558 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-metrics-server-audit-profiles\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526594 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9xv\" (UniqueName: \"kubernetes.io/projected/fd59f213-a474-4fb6-a994-c3dd365fe467-kube-api-access-tk9xv\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526621 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-client-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526699 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526668 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd59f213-a474-4fb6-a994-c3dd365fe467-audit-log\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526840 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526818 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-tls\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526883 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526839 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-client-certs\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.526883 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.526864 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.595633 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.595603 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw"] Nov 02 19:48:44.598314 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.598290 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.627654 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627605 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9xv\" (UniqueName: \"kubernetes.io/projected/fd59f213-a474-4fb6-a994-c3dd365fe467-kube-api-access-tk9xv\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627654 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627641 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-client-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627814 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627660 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd59f213-a474-4fb6-a994-c3dd365fe467-audit-log\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627814 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627776 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-tls\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627911 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627818 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-client-certs\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627911 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627860 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627999 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627969 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-metrics-server-audit-profiles\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.627999 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.627988 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd59f213-a474-4fb6-a994-c3dd365fe467-audit-log\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.628510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.628490 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.629090 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.629067 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd59f213-a474-4fb6-a994-c3dd365fe467-metrics-server-audit-profiles\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.630198 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.630177 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-client-certs\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.630339 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.630321 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-secret-metrics-server-tls\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.630398 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.630381 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd59f213-a474-4fb6-a994-c3dd365fe467-client-ca-bundle\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.652638 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652615 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 19:48:44.652638 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652629 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 19:48:44.652792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652629 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 19:48:44.652792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652709 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 19:48:44.652792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652755 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 19:48:44.652792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652758 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2qbjb84dghmgp\"" Nov 02 19:48:44.652933 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.652794 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t9g6q\"" Nov 02 19:48:44.658966 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.658944 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw"] Nov 02 19:48:44.683194 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.683169 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9xv\" (UniqueName: \"kubernetes.io/projected/fd59f213-a474-4fb6-a994-c3dd365fe467-kube-api-access-tk9xv\") pod \"metrics-server-995647d55-2ckjz\" (UID: \"fd59f213-a474-4fb6-a994-c3dd365fe467\") " pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.689914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.689895 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:44.728833 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.728806 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.728973 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.728838 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9e00b19-2153-4977-8126-4a5268f7252a-metrics-client-ca\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.728973 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.728860 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-grpc-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.728973 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.728879 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e00b19-2153-4977-8126-4a5268f7252a-tmp\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.728973 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.728942 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xdh\" (UniqueName: \"kubernetes.io/projected/a9e00b19-2153-4977-8126-4a5268f7252a-kube-api-access-j8xdh\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.729118 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.729060 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.729175 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.729157 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.729267 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.729248 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.729324 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.729308 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830353 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830391 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830409 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9e00b19-2153-4977-8126-4a5268f7252a-metrics-client-ca\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830427 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-grpc-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830451 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e00b19-2153-4977-8126-4a5268f7252a-tmp\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830473 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xdh\" (UniqueName: \"kubernetes.io/projected/a9e00b19-2153-4977-8126-4a5268f7252a-kube-api-access-j8xdh\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830501 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830555 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830537 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.830968 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.830584 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.831373 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.831325 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e00b19-2153-4977-8126-4a5268f7252a-tmp\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.831482 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.831442 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9e00b19-2153-4977-8126-4a5268f7252a-metrics-client-ca\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833187 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833169 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-grpc-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833167 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833414 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833391 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833505 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-tls\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833567 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833542 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.833640 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.833625 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9e00b19-2153-4977-8126-4a5268f7252a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.884459 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.884400 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xdh\" (UniqueName: \"kubernetes.io/projected/a9e00b19-2153-4977-8126-4a5268f7252a-kube-api-access-j8xdh\") pod \"thanos-querier-57d6d89c5c-4ccnw\" (UID: \"a9e00b19-2153-4977-8126-4a5268f7252a\") " pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:44.917292 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:44.917267 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:45.096410 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.096378 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-995647d55-2ckjz"] Nov 02 19:48:45.099355 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:45.099331 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd59f213_a474_4fb6_a994_c3dd365fe467.slice/crio-0855e78dc4de14876f7d1e0672248e9a1ae61e38f51349db97e4a55bacaff2c9 WatchSource:0}: Error finding container 0855e78dc4de14876f7d1e0672248e9a1ae61e38f51349db97e4a55bacaff2c9: Status 404 returned error can't find the container with id 0855e78dc4de14876f7d1e0672248e9a1ae61e38f51349db97e4a55bacaff2c9 Nov 02 19:48:45.279458 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.279427 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw"] Nov 02 19:48:45.282128 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:45.282098 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e00b19_2153_4977_8126_4a5268f7252a.slice/crio-c1cde47081ddbf419d82c0283452ad2c79f1287f2afaaaed1619c66a289c202b WatchSource:0}: Error finding container c1cde47081ddbf419d82c0283452ad2c79f1287f2afaaaed1619c66a289c202b: Status 404 returned error can't find the container with id c1cde47081ddbf419d82c0283452ad2c79f1287f2afaaaed1619c66a289c202b Nov 02 19:48:45.426833 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.426806 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:45.427157 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.426899 5136 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:48:45.431385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.431370 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q2q9h" Nov 02 19:48:45.629822 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.629745 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"c1cde47081ddbf419d82c0283452ad2c79f1287f2afaaaed1619c66a289c202b"} Nov 02 19:48:45.631028 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:45.630998 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" event={"ID":"fd59f213-a474-4fb6-a994-c3dd365fe467","Type":"ContainerStarted","Data":"0855e78dc4de14876f7d1e0672248e9a1ae61e38f51349db97e4a55bacaff2c9"} Nov 02 19:48:46.383986 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.383950 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:48:46.386863 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.386839 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.441046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.441012 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 19:48:46.441046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.441011 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 19:48:46.441046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.441013 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 19:48:46.484626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.484596 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 19:48:46.484626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.484620 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 19:48:46.484886 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.484603 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 19:48:46.485932 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.485892 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 19:48:46.532869 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.532839 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x9fws\"" Nov 02 19:48:46.542801 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.542757 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.542964 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.542863 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.542964 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.542903 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.543079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.542985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.543079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.543007 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgv7m\" (UniqueName: \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.543079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.543037 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.543240 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.543131 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.543240 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.543155 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.584643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.584452 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 19:48:46.585119 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.585092 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:48:46.646530 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646437 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646530 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646495 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646530 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646514 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646546 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646570 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646596 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646623 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.646857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.646640 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgv7m\" (UniqueName: \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.647239 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.647218 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.648236 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.648112 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.648609 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.648587 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.648723 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.648637 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.648828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.648792 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.652715 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.652691 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.662224 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.662199 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.702551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.702518 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgv7m\" (UniqueName: \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\") pod \"console-7db5c775fc-vdrjd\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:46.999355 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:46.999320 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:47.106616 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:47.106589 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:48:47.370944 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:47.370838 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:48:47.376175 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:47.376148 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da74eee_a150_4a0a_92ba_6950259f1fca.slice/crio-5ddf05ab71b3adc454eb243fcc421f79a6e4697c7ae532f9976b632a89febbfd WatchSource:0}: Error finding container 5ddf05ab71b3adc454eb243fcc421f79a6e4697c7ae532f9976b632a89febbfd: Status 404 returned error can't find the container with id 5ddf05ab71b3adc454eb243fcc421f79a6e4697c7ae532f9976b632a89febbfd Nov 02 19:48:47.647541 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:47.641050 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db5c775fc-vdrjd" event={"ID":"9da74eee-a150-4a0a-92ba-6950259f1fca","Type":"ContainerStarted","Data":"5ddf05ab71b3adc454eb243fcc421f79a6e4697c7ae532f9976b632a89febbfd"} Nov 02 19:48:47.759656 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:47.758703 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:48:47.759656 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:47.758998 5136 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:48:48.647336 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:48.647298 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" event={"ID":"fd59f213-a474-4fb6-a994-c3dd365fe467","Type":"ContainerStarted","Data":"5cdd81f2835b799f0e5ae18519b7ab56f565b9be2f9d9adfff0272dd3c8f0daf"} Nov 02 19:48:48.649830 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:48.649803 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"d3c1728e05e37bc631e4a6c5870b21d47498f19ce939e92a3f643f1d9839ffd1"} Nov 02 19:48:48.650169 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:48.649873 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"6de3a215ba2b3fda32fe2a50ca3703eb4bf6fd0e3d01a263548ae90c13f88002"} Nov 02 19:48:48.650169 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:48.649911 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"1c35efc9d301bab5228a1f86ce02267a33e3470b1e15736f3dcb4f84b29ac557"} Nov 02 19:48:48.756515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:48.756459 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" podStartSLOduration=2.278504132 podStartE2EDuration="4.75644506s" totalImagesPullingTime="2.477940928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:48.755942084 +0000 UTC m=+12.362486582" watchObservedRunningTime="2026-11-02 19:48:48.75644506 +0000 UTC m=+12.362989558" Nov 02 19:48:49.446530 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:49.446498 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:49.657376 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:49.657338 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"31c4ce302db8e69ec36b7bf31b7abd6086869438de2baf66427cfeea553cab85"} Nov 02 19:48:49.657876 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:49.657420 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"7712ee7da5467477d65b401978ae25043aa4914a8701dd96378cae8511ccfde1"} Nov 02 19:48:49.657876 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:49.657465 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" event={"ID":"a9e00b19-2153-4977-8126-4a5268f7252a","Type":"ContainerStarted","Data":"dba6d0be65c749662d1d048e6d0034808a8f8b5dda2868035900d4a00d5de9d6"} Nov 02 19:48:49.804369 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:49.804275 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" podStartSLOduration=2.10527999 podStartE2EDuration="5.80426152s" totalImagesPullingTime="3.69898153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:49.770146065 +0000 UTC m=+13.376690564" watchObservedRunningTime="2026-11-02 19:48:49.80426152 +0000 UTC m=+13.410806015" Nov 02 19:48:50.661764 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:50.661719 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db5c775fc-vdrjd" event={"ID":"9da74eee-a150-4a0a-92ba-6950259f1fca","Type":"ContainerStarted","Data":"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f"} Nov 02 19:48:50.662286 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:50.662265 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:50.805196 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:50.805149 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7db5c775fc-vdrjd" podStartSLOduration=1.5876170790000002 podStartE2EDuration="4.805136541s" totalImagesPullingTime="3.217519462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:50.805025817 +0000 UTC m=+14.411570314" watchObservedRunningTime="2026-11-02 19:48:50.805136541 +0000 UTC m=+14.411681031" Nov 02 19:48:51.003226 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:48:51.003195 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:48:51.217216 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:51.217190 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wfdgc" Nov 02 19:48:55.086294 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:55.086257 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:48:56.405706 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.405676 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:48:56.505205 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.505177 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:48:56.558247 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.558215 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:48:56.558403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.558300 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.625758 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625716 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.625897 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625764 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.625897 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625783 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.625897 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625872 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.626002 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625916 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.626002 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.625984 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfjw\" (UniqueName: \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.626062 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.626029 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.626062 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.626055 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.670490 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.670432 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57d6d89c5c-4ccnw" Nov 02 19:48:56.727211 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727188 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727341 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727251 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfjw\" (UniqueName: \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727341 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727304 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727532 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727506 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727623 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727602 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727677 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727662 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727752 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727698 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.727803 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.727749 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.728181 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.728156 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.728296 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.728183 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.728296 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.728233 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.728296 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.728248 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.728461 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.728440 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.729948 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.729930 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.730047 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.730030 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.780881 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.780858 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfjw\" (UniqueName: \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\") pod \"console-76bfcf84c7-kgn6j\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:56.881676 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:56.881645 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:48:57.000213 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:57.000187 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:48:57.267767 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:57.267684 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:48:57.270900 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:48:57.270867 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598d4c65_6138_4b27_b8e9_22bd3c806b72.slice/crio-5dc253214f343bbe5beda58eddca91c369ce392616a41da52404bc14013322e1 WatchSource:0}: Error finding container 5dc253214f343bbe5beda58eddca91c369ce392616a41da52404bc14013322e1: Status 404 returned error can't find the container with id 5dc253214f343bbe5beda58eddca91c369ce392616a41da52404bc14013322e1 Nov 02 19:48:57.687025 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:57.686938 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bfcf84c7-kgn6j" event={"ID":"598d4c65-6138-4b27-b8e9-22bd3c806b72","Type":"ContainerStarted","Data":"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166"} Nov 02 19:48:57.687025 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:57.686992 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bfcf84c7-kgn6j" event={"ID":"598d4c65-6138-4b27-b8e9-22bd3c806b72","Type":"ContainerStarted","Data":"5dc253214f343bbe5beda58eddca91c369ce392616a41da52404bc14013322e1"} Nov 02 19:48:57.796811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:57.796763 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76bfcf84c7-kgn6j" podStartSLOduration=1.796751402 podStartE2EDuration="1.796751402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:57.79627897 +0000 UTC m=+21.402823470" watchObservedRunningTime="2026-11-02 19:48:57.796751402 +0000 UTC m=+21.403295893" Nov 02 19:48:59.131931 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:48:59.131903 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bff7p" Nov 02 19:49:06.882277 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:06.882232 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:49:06.882866 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:06.882327 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:49:06.886900 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:06.886876 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:49:07.724954 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:07.724928 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:49:09.451963 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:09.451931 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:49:09.455597 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:09.455576 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-995647d55-2ckjz" Nov 02 19:49:17.109570 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:17.109540 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nvcwl" Nov 02 19:49:21.002326 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:49:21.002286 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:49:21.425191 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.425088 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7db5c775fc-vdrjd" podUID="9da74eee-a150-4a0a-92ba-6950259f1fca" containerName="console" containerID="cri-o://12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f" gracePeriod=15 Nov 02 19:49:21.688809 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.688749 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:49:21.766323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.766296 5136 generic.go:356] "Generic (PLEG): container finished" podID="9da74eee-a150-4a0a-92ba-6950259f1fca" containerID="12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f" exitCode=0 Nov 02 19:49:21.766456 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.766338 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db5c775fc-vdrjd" Nov 02 19:49:21.766456 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.766384 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db5c775fc-vdrjd" event={"ID":"9da74eee-a150-4a0a-92ba-6950259f1fca","Type":"ContainerDied","Data":"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f"} Nov 02 19:49:21.766569 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.766461 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db5c775fc-vdrjd" event={"ID":"9da74eee-a150-4a0a-92ba-6950259f1fca","Type":"ContainerDied","Data":"5ddf05ab71b3adc454eb243fcc421f79a6e4697c7ae532f9976b632a89febbfd"} Nov 02 19:49:21.766569 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.766482 5136 scope.go:122] "RemoveContainer" containerID="12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f" Nov 02 19:49:21.774130 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.774107 5136 scope.go:122] "RemoveContainer" containerID="12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f" Nov 02 19:49:21.774383 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:49:21.774361 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f\": container with ID starting with 12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f not found: ID does not exist" containerID="12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f" Nov 02 19:49:21.774449 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.774391 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f"} err="failed to get container status \"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f\": rpc error: code = NotFound desc = could not find container \"12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f\": container with ID starting with 12af930ee2cbf029a373cb9937059b652e32ac225ed2e462e4c19308d4793b8f not found: ID does not exist" Nov 02 19:49:21.808553 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808533 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808661 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808566 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808661 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808587 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808661 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808639 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\" (UniqueName: \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808662 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808686 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808771 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.808802 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.808796 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\") pod \"9da74eee-a150-4a0a-92ba-6950259f1fca\" (UID: \"9da74eee-a150-4a0a-92ba-6950259f1fca\") " Nov 02 19:49:21.809075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.809048 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:21.809218 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.809190 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:21.809331 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.809299 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:21.809399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.809379 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:21.809457 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.809394 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:21.811037 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.811014 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:21.811138 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.811076 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "kube-api-access-zgv7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:21.811287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.811263 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config" pod "9da74eee-a150-4a0a-92ba-6950259f1fca" (UID: "9da74eee-a150-4a0a-92ba-6950259f1fca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:21.910006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.909983 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgv7m\" (UniqueName: \"kubernetes.io/projected/9da74eee-a150-4a0a-92ba-6950259f1fca-kube-api-access-zgv7m\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910004 5136 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da74eee-a150-4a0a-92ba-6950259f1fca-tmp\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910019 5136 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-console-config\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910032 5136 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-oauth-config\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910040 5136 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-trusted-ca-bundle\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910048 5136 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910057 5136 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9da74eee-a150-4a0a-92ba-6950259f1fca-console-serving-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:21.910143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:21.910066 5136 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9da74eee-a150-4a0a-92ba-6950259f1fca-oauth-serving-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:22.178644 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:22.178615 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:49:22.183515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:22.183492 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7db5c775fc-vdrjd"] Nov 02 19:49:22.543323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:49:22.543290 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da74eee-a150-4a0a-92ba-6950259f1fca" path="/var/lib/kubelet/pods/9da74eee-a150-4a0a-92ba-6950259f1fca/volumes" Nov 02 19:49:51.001979 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:49:51.001941 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:50:14.382551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.382515 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-595b69756b-74lj9"] Nov 02 19:50:14.383171 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.382912 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9da74eee-a150-4a0a-92ba-6950259f1fca" containerName="console" Nov 02 19:50:14.383171 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.382931 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da74eee-a150-4a0a-92ba-6950259f1fca" containerName="console" Nov 02 19:50:14.383171 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.383028 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da74eee-a150-4a0a-92ba-6950259f1fca" containerName="console" Nov 02 19:50:14.385436 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.385416 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.438913 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.438884 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b69756b-74lj9"] Nov 02 19:50:14.467044 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467022 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467174 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467051 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-service-ca\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467174 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467072 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-console-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467174 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467087 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65712f8c-2ddf-4276-88a5-30a0726202f7-tmp\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467174 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467146 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-trusted-ca-bundle\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467329 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467315 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-oauth-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467358 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-oauth-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.467404 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.467377 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrll\" (UniqueName: \"kubernetes.io/projected/65712f8c-2ddf-4276-88a5-30a0726202f7-kube-api-access-svrll\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568033 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568008 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-oauth-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568163 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568037 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrll\" (UniqueName: \"kubernetes.io/projected/65712f8c-2ddf-4276-88a5-30a0726202f7-kube-api-access-svrll\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568163 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568058 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568282 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568259 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-service-ca\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568340 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568324 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-console-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568402 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568351 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65712f8c-2ddf-4276-88a5-30a0726202f7-tmp\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568402 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568388 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-trusted-ca-bundle\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568506 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568429 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-oauth-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.568827 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.568770 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65712f8c-2ddf-4276-88a5-30a0726202f7-tmp\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.569092 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.569072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-oauth-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.569169 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.569106 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-console-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.569169 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.569118 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-service-ca\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.569456 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.569433 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65712f8c-2ddf-4276-88a5-30a0726202f7-trusted-ca-bundle\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.570565 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.570548 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-serving-cert\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.570639 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.570623 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65712f8c-2ddf-4276-88a5-30a0726202f7-console-oauth-config\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.623035 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.623006 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrll\" (UniqueName: \"kubernetes.io/projected/65712f8c-2ddf-4276-88a5-30a0726202f7-kube-api-access-svrll\") pod \"console-595b69756b-74lj9\" (UID: \"65712f8c-2ddf-4276-88a5-30a0726202f7\") " pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:14.696268 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:14.696244 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:15.054009 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:15.053988 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b69756b-74lj9"] Nov 02 19:50:15.058198 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:50:15.058165 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65712f8c_2ddf_4276_88a5_30a0726202f7.slice/crio-886d4e6fffcf93085fea67ef2cb01ca42542952c64e1065dd9c3000b166b76cf WatchSource:0}: Error finding container 886d4e6fffcf93085fea67ef2cb01ca42542952c64e1065dd9c3000b166b76cf: Status 404 returned error can't find the container with id 886d4e6fffcf93085fea67ef2cb01ca42542952c64e1065dd9c3000b166b76cf Nov 02 19:50:15.942301 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:15.942268 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b69756b-74lj9" event={"ID":"65712f8c-2ddf-4276-88a5-30a0726202f7","Type":"ContainerStarted","Data":"f93f4fe5450860095932de8e47cd0c89904ebfb805d2344dd00efb46c3192635"} Nov 02 19:50:15.942823 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:15.942336 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b69756b-74lj9" event={"ID":"65712f8c-2ddf-4276-88a5-30a0726202f7","Type":"ContainerStarted","Data":"886d4e6fffcf93085fea67ef2cb01ca42542952c64e1065dd9c3000b166b76cf"} Nov 02 19:50:16.052079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:16.052027 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-595b69756b-74lj9" podStartSLOduration=2.052013153 podStartE2EDuration="2.052013153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:16.051266752 +0000 UTC m=+99.657811253" watchObservedRunningTime="2026-11-02 19:50:16.052013153 +0000 UTC m=+99.658557651" Nov 02 19:50:21.002167 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:50:21.002088 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:50:24.697224 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:24.697187 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:24.697683 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:24.697259 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:24.701894 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:24.701871 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:24.975271 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:24.975198 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595b69756b-74lj9" Nov 02 19:50:25.109108 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:25.109077 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:50:50.128594 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.128559 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76bfcf84c7-kgn6j" podUID="598d4c65-6138-4b27-b8e9-22bd3c806b72" containerName="console" containerID="cri-o://3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166" gracePeriod=15 Nov 02 19:50:50.356351 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.356326 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:50:50.442294 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442240 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442294 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442288 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442452 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442320 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\" (UniqueName: \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442452 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442354 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442452 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442386 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442602 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442545 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442654 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442604 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\" (UniqueName: \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442654 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442640 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\") pod \"598d4c65-6138-4b27-b8e9-22bd3c806b72\" (UID: \"598d4c65-6138-4b27-b8e9-22bd3c806b72\") " Nov 02 19:50:50.442777 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442672 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:50:50.442855 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442834 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:50:50.442908 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442883 5136 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-config\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.442959 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442896 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:50:50.442959 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.442919 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:50:50.443240 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.443215 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:50:50.444571 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.444540 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "kube-api-access-rbfjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:50:50.444571 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.444539 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:50:50.444694 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.444552 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert" pod "598d4c65-6138-4b27-b8e9-22bd3c806b72" (UID: "598d4c65-6138-4b27-b8e9-22bd3c806b72"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:50:50.544098 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544074 5136 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-oauth-config\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544098 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544100 5136 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-trusted-ca-bundle\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544115 5136 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544128 5136 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598d4c65-6138-4b27-b8e9-22bd3c806b72-tmp\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544148 5136 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/598d4c65-6138-4b27-b8e9-22bd3c806b72-oauth-serving-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544159 5136 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/598d4c65-6138-4b27-b8e9-22bd3c806b72-console-serving-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:50.544280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:50.544172 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfjw\" (UniqueName: \"kubernetes.io/projected/598d4c65-6138-4b27-b8e9-22bd3c806b72-kube-api-access-rbfjw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:50:51.002400 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:50:51.002362 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:50:51.057553 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.057524 5136 generic.go:356] "Generic (PLEG): container finished" podID="598d4c65-6138-4b27-b8e9-22bd3c806b72" containerID="3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166" exitCode=0 Nov 02 19:50:51.057700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.057572 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bfcf84c7-kgn6j" Nov 02 19:50:51.057700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.057610 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bfcf84c7-kgn6j" event={"ID":"598d4c65-6138-4b27-b8e9-22bd3c806b72","Type":"ContainerDied","Data":"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166"} Nov 02 19:50:51.057700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.057668 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bfcf84c7-kgn6j" event={"ID":"598d4c65-6138-4b27-b8e9-22bd3c806b72","Type":"ContainerDied","Data":"5dc253214f343bbe5beda58eddca91c369ce392616a41da52404bc14013322e1"} Nov 02 19:50:51.057700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.057691 5136 scope.go:122] "RemoveContainer" containerID="3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166" Nov 02 19:50:51.065187 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.065169 5136 scope.go:122] "RemoveContainer" containerID="3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166" Nov 02 19:50:51.065437 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:50:51.065417 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166\": container with ID starting with 3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166 not found: ID does not exist" containerID="3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166" Nov 02 19:50:51.065500 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.065445 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166"} err="failed to get container status \"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166\": rpc error: code = NotFound desc = could not find container \"3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166\": container with ID starting with 3158ce90c3cd51c868b9a849a83d5558ff582914d88cf0fbc01d2856d6d89166 not found: ID does not exist" Nov 02 19:50:51.212081 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.212050 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:50:51.259464 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:51.259394 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76bfcf84c7-kgn6j"] Nov 02 19:50:52.543918 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:50:52.543887 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598d4c65-6138-4b27-b8e9-22bd3c806b72" path="/var/lib/kubelet/pods/598d4c65-6138-4b27-b8e9-22bd3c806b72/volumes" Nov 02 19:51:21.002341 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:51:21.002304 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:51:51.002434 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:51:51.002359 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:52:21.002451 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:52:21.002407 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:52:26.371821 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.371791 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8"] Nov 02 19:52:26.374093 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.372050 5136 cpu_manager.go:414] "containerMap: removing container" podUID="598d4c65-6138-4b27-b8e9-22bd3c806b72" containerName="console" Nov 02 19:52:26.374093 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.372060 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598d4c65-6138-4b27-b8e9-22bd3c806b72" containerName="console" Nov 02 19:52:26.374093 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.372143 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="598d4c65-6138-4b27-b8e9-22bd3c806b72" containerName="console" Nov 02 19:52:26.375030 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.374998 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.475389 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.475352 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:52:26.475389 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.475367 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:52:26.475567 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.475441 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rn4dc\"" Nov 02 19:52:26.475923 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.475906 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8"] Nov 02 19:52:26.510005 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.509979 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ltz\" (UniqueName: \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.510175 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.510048 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.510175 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.510101 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.610971 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.610946 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.611094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.610977 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.611094 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.611020 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ltz\" (UniqueName: \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.611307 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.611288 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.611367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.611330 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.669001 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.668934 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ltz\" (UniqueName: \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.691882 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.691843 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:26.724398 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.724369 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:52:26.727846 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.727821 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.781080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781044 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 19:52:26.781080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781045 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 19:52:26.781080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781072 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:52:26.781333 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781201 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:52:26.781333 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781213 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 19:52:26.781333 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.781266 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqzr\"" Nov 02 19:52:26.783325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.783306 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 19:52:26.799285 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.799263 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:52:26.812713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.812690 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.812836 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.812749 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.812879 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.812862 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.812912 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.812889 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.812958 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.812912 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.813059 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.813041 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.813102 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.813068 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzt7k\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914046 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914011 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914055 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914096 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914120 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914144 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914199 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914482 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914222 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzt7k\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.914857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.914829 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.916546 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.916517 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.916643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.916621 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.916748 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.916712 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.916872 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.916856 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.973819 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.973797 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:26.976323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:26.976300 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzt7k\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\") pod \"istiod-openshift-gateway-7876bfcb74-9tscf\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:27.040923 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:27.040901 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:27.058215 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:27.058195 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8"] Nov 02 19:52:27.060065 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:52:27.060043 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e070981_855b_4012_b3f9_d7a12eb0390f.slice/crio-a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e WatchSource:0}: Error finding container a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e: Status 404 returned error can't find the container with id a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e Nov 02 19:52:27.374061 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:27.373986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" event={"ID":"7e070981-855b-4012-b3f9-d7a12eb0390f","Type":"ContainerStarted","Data":"a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e"} Nov 02 19:52:27.402067 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:27.402045 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:52:27.404302 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:52:27.404274 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba7ff3_b4b2_4369_84dc_e59178de6a86.slice/crio-7dd345cc5ecc5ad61fafd7432583343a107c87bcb9250cfad4b823fa5013d185 WatchSource:0}: Error finding container 7dd345cc5ecc5ad61fafd7432583343a107c87bcb9250cfad4b823fa5013d185: Status 404 returned error can't find the container with id 7dd345cc5ecc5ad61fafd7432583343a107c87bcb9250cfad4b823fa5013d185 Nov 02 19:52:28.378095 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:28.378061 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" event={"ID":"66ba7ff3-b4b2-4369-84dc-e59178de6a86","Type":"ContainerStarted","Data":"7dd345cc5ecc5ad61fafd7432583343a107c87bcb9250cfad4b823fa5013d185"} Nov 02 19:52:31.720018 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:31.719979 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:31.720381 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:31.720061 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:32.395510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:32.394979 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" event={"ID":"66ba7ff3-b4b2-4369-84dc-e59178de6a86","Type":"ContainerStarted","Data":"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9"} Nov 02 19:52:32.395510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:32.395178 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:32.396986 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:32.396946 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-9tscf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:52:32.397118 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:32.397011 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:52:32.592214 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:32.592151 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" podStartSLOduration=2.279407532 podStartE2EDuration="6.592133234s" totalImagesPullingTime="4.312725702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:32.553403632 +0000 UTC m=+236.159948144" watchObservedRunningTime="2026-11-02 19:52:32.592133234 +0000 UTC m=+236.198677736" Nov 02 19:52:33.400187 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:33.400151 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-9tscf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:52:33.400678 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:33.400214 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:52:34.404551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:34.404520 5136 generic.go:356] "Generic (PLEG): container finished" podID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerID="0f88ac76042bdf8b31e8bea99fe694b54de0e9669c9a49029141cacbae3ff949" exitCode=0 Nov 02 19:52:34.405043 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:34.404585 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" event={"ID":"7e070981-855b-4012-b3f9-d7a12eb0390f","Type":"ContainerDied","Data":"0f88ac76042bdf8b31e8bea99fe694b54de0e9669c9a49029141cacbae3ff949"} Nov 02 19:52:36.400158 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:36.400127 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:52:38.419945 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:38.419868 5136 generic.go:356] "Generic (PLEG): container finished" podID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerID="a5580d6509ef96914642db6718d66c7f1517bdfd4e5a99ab0a4095f14b2c19e6" exitCode=0 Nov 02 19:52:38.420305 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:38.419955 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" event={"ID":"7e070981-855b-4012-b3f9-d7a12eb0390f","Type":"ContainerDied","Data":"a5580d6509ef96914642db6718d66c7f1517bdfd4e5a99ab0a4095f14b2c19e6"} Nov 02 19:52:45.448179 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:45.448153 5136 generic.go:356] "Generic (PLEG): container finished" podID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerID="6874cffe2745f310a1062ff194f1da918d5e2f299315a47bc225bcba3be2d5c3" exitCode=0 Nov 02 19:52:45.448492 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:45.448212 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" event={"ID":"7e070981-855b-4012-b3f9-d7a12eb0390f","Type":"ContainerDied","Data":"6874cffe2745f310a1062ff194f1da918d5e2f299315a47bc225bcba3be2d5c3"} Nov 02 19:52:46.569800 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.569773 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:46.668827 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.668793 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\") pod \"7e070981-855b-4012-b3f9-d7a12eb0390f\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " Nov 02 19:52:46.668963 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.668843 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\") pod \"7e070981-855b-4012-b3f9-d7a12eb0390f\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " Nov 02 19:52:46.668963 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.668881 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\" (UniqueName: \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\") pod \"7e070981-855b-4012-b3f9-d7a12eb0390f\" (UID: \"7e070981-855b-4012-b3f9-d7a12eb0390f\") " Nov 02 19:52:46.669463 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.669437 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle" pod "7e070981-855b-4012-b3f9-d7a12eb0390f" (UID: "7e070981-855b-4012-b3f9-d7a12eb0390f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:46.671302 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.671276 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz" pod "7e070981-855b-4012-b3f9-d7a12eb0390f" (UID: "7e070981-855b-4012-b3f9-d7a12eb0390f"). InnerVolumeSpecName "kube-api-access-n2ltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:52:46.673618 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.673597 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util" pod "7e070981-855b-4012-b3f9-d7a12eb0390f" (UID: "7e070981-855b-4012-b3f9-d7a12eb0390f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:46.769425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.769362 5136 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-util\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:46.769425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.769385 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ltz\" (UniqueName: \"kubernetes.io/projected/7e070981-855b-4012-b3f9-d7a12eb0390f-kube-api-access-n2ltz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:46.769425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:46.769396 5136 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e070981-855b-4012-b3f9-d7a12eb0390f-bundle\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:47.456282 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:47.456246 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" event={"ID":"7e070981-855b-4012-b3f9-d7a12eb0390f","Type":"ContainerDied","Data":"a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e"} Nov 02 19:52:47.456282 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:47.456286 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9jtz8" Nov 02 19:52:47.456488 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:52:47.456327 5136 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b94101340c05ce3f13d562dcb6fbfa24ca2a0e6fe13421707abd4e115f721e" Nov 02 19:52:51.002506 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:52:51.002474 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:53:13.877288 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877255 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd"] Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877551 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="pull" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877561 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="pull" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877571 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="extract" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877576 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="extract" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877594 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="util" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877599 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="util" Nov 02 19:53:13.877775 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.877653 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e070981-855b-4012-b3f9-d7a12eb0390f" containerName="extract" Nov 02 19:53:13.941290 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:13.941267 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.040373 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.040341 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd"] Nov 02 19:53:14.045997 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.045977 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:53:14.046080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.046033 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:53:14.046080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.046071 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4tm9h\"" Nov 02 19:53:14.046164 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.046080 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:53:14.047579 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.047545 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:53:14.047630 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.047610 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:53:14.060986 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.060963 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.061057 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.060993 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgpw\" (UniqueName: \"kubernetes.io/projected/fab4a53c-1590-4db0-b449-b2e31b5520a5-kube-api-access-xtgpw\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.061057 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.061016 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.061149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.061136 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.061184 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.061158 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.061222 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.061185 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.161859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161781 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.161859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161821 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.162075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161860 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.162075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161918 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.162075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161956 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgpw\" (UniqueName: \"kubernetes.io/projected/fab4a53c-1590-4db0-b449-b2e31b5520a5-kube-api-access-xtgpw\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.162075 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.161989 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.162262 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.162236 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.164308 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.164285 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.164414 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.164388 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.164454 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.164421 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fab4a53c-1590-4db0-b449-b2e31b5520a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.164502 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.164486 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab4a53c-1590-4db0-b449-b2e31b5520a5-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.217156 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.217126 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgpw\" (UniqueName: \"kubernetes.io/projected/fab4a53c-1590-4db0-b449-b2e31b5520a5-kube-api-access-xtgpw\") pod \"opendatahub-operator-controller-manager-55d64b46df-45hdd\" (UID: \"fab4a53c-1590-4db0-b449-b2e31b5520a5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.276881 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.276828 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:14.822180 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:14.822158 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd"] Nov 02 19:53:14.824322 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:53:14.824286 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab4a53c_1590_4db0_b449_b2e31b5520a5.slice/crio-1b62b8d064c295b2f017cae90dfbde70b6ffcd9995b51f04c5679b73c9ebe529 WatchSource:0}: Error finding container 1b62b8d064c295b2f017cae90dfbde70b6ffcd9995b51f04c5679b73c9ebe529: Status 404 returned error can't find the container with id 1b62b8d064c295b2f017cae90dfbde70b6ffcd9995b51f04c5679b73c9ebe529 Nov 02 19:53:15.560178 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:15.560146 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" event={"ID":"fab4a53c-1590-4db0-b449-b2e31b5520a5","Type":"ContainerStarted","Data":"1b62b8d064c295b2f017cae90dfbde70b6ffcd9995b51f04c5679b73c9ebe529"} Nov 02 19:53:18.578113 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:18.578070 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" event={"ID":"fab4a53c-1590-4db0-b449-b2e31b5520a5","Type":"ContainerStarted","Data":"f12ede5f468660c9c1af42a1fcee1d9162d86c4c6bcd224bf4561f33b60128e1"} Nov 02 19:53:19.582675 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:19.582642 5136 generic.go:356] "Generic (PLEG): container finished" podID="fab4a53c-1590-4db0-b449-b2e31b5520a5" containerID="f12ede5f468660c9c1af42a1fcee1d9162d86c4c6bcd224bf4561f33b60128e1" exitCode=0 Nov 02 19:53:19.583356 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:19.582689 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" event={"ID":"fab4a53c-1590-4db0-b449-b2e31b5520a5","Type":"ContainerDied","Data":"f12ede5f468660c9c1af42a1fcee1d9162d86c4c6bcd224bf4561f33b60128e1"} Nov 02 19:53:20.587523 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:20.587488 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" event={"ID":"fab4a53c-1590-4db0-b449-b2e31b5520a5","Type":"ContainerStarted","Data":"f91b38db8f8e8fb0ba034eace7beb792c5cc724672daca8bf84363e68e412f0a"} Nov 02 19:53:20.587908 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:20.587603 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:20.817625 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:20.817531 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" podStartSLOduration=2.943381096 podStartE2EDuration="7.817516937s" totalImagesPullingTime="4.413241319s" totalInitContainerRuntime="460.894522ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:20.769925185 +0000 UTC m=+284.376469682" watchObservedRunningTime="2026-11-02 19:53:20.817516937 +0000 UTC m=+284.424061428" Nov 02 19:53:21.001518 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:53:21.001484 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:53:31.594830 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:31.594798 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-45hdd" Nov 02 19:53:40.239744 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.239696 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg"] Nov 02 19:53:40.244500 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.244480 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:40.295281 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.295257 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:53:40.342759 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.342718 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 19:53:40.342759 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.342745 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mxj5l\"" Nov 02 19:53:40.342932 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.342810 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:53:40.343781 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.343765 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg"] Nov 02 19:53:40.376978 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.376952 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pn64\" (UniqueName: \"kubernetes.io/projected/ba3a6e88-66b2-4b55-a673-6260a7cbbe1a-kube-api-access-7pn64\") pod \"dns-operator-controller-manager-8557cbdb84-7cwmg\" (UID: \"ba3a6e88-66b2-4b55-a673-6260a7cbbe1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:40.478214 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.478186 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pn64\" (UniqueName: \"kubernetes.io/projected/ba3a6e88-66b2-4b55-a673-6260a7cbbe1a-kube-api-access-7pn64\") pod \"dns-operator-controller-manager-8557cbdb84-7cwmg\" (UID: \"ba3a6e88-66b2-4b55-a673-6260a7cbbe1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:40.543942 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.543878 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pn64\" (UniqueName: \"kubernetes.io/projected/ba3a6e88-66b2-4b55-a673-6260a7cbbe1a-kube-api-access-7pn64\") pod \"dns-operator-controller-manager-8557cbdb84-7cwmg\" (UID: \"ba3a6e88-66b2-4b55-a673-6260a7cbbe1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:40.545926 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.545901 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd"] Nov 02 19:53:40.548186 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.548168 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.555287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.555265 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:40.603500 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.603467 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd"] Nov 02 19:53:40.604129 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.603934 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d2pr6\"" Nov 02 19:53:40.604129 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.604005 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 19:53:40.604129 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.604073 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 19:53:40.680059 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.680032 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4m4\" (UniqueName: \"kubernetes.io/projected/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-kube-api-access-2z4m4\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.680206 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.680084 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.680206 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.680178 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.781410 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.781381 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.781605 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.781417 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.781605 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.781452 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4m4\" (UniqueName: \"kubernetes.io/projected/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-kube-api-access-2z4m4\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.781605 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:53:40.781551 5136 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 02 19:53:40.781824 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:53:40.781619 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert podName:0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff nodeName:}" failed. No retries permitted until 2026-11-02 19:53:41.281601191 +0000 UTC m=+304.888145690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-pppkd" (UID: "0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff") : secret "plugin-serving-cert" not found Nov 02 19:53:40.782166 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.782141 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.836015 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.835956 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4m4\" (UniqueName: \"kubernetes.io/projected/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-kube-api-access-2z4m4\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:40.921791 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.921766 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg"] Nov 02 19:53:40.924248 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:53:40.924220 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3a6e88_66b2_4b55_a673_6260a7cbbe1a.slice/crio-4b825f4726d81fb553c2cdc10977122f6220bf0cc298cd226b1ccbce62a579dd WatchSource:0}: Error finding container 4b825f4726d81fb553c2cdc10977122f6220bf0cc298cd226b1ccbce62a579dd: Status 404 returned error can't find the container with id 4b825f4726d81fb553c2cdc10977122f6220bf0cc298cd226b1ccbce62a579dd Nov 02 19:53:40.926523 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:40.926507 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:53:41.285989 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:41.285953 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:41.288797 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:41.288770 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pppkd\" (UID: \"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:41.472892 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:41.472861 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" Nov 02 19:53:41.662651 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:41.662575 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" event={"ID":"ba3a6e88-66b2-4b55-a673-6260a7cbbe1a","Type":"ContainerStarted","Data":"4b825f4726d81fb553c2cdc10977122f6220bf0cc298cd226b1ccbce62a579dd"} Nov 02 19:53:41.834177 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:41.834148 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd"] Nov 02 19:53:41.835813 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:53:41.835783 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c5f0991_ec7e_4cef_ac8b_43495b3ba3ff.slice/crio-5cd7cc152fcf91331ae5b14f4e2e0c5c454c43037d53dd4ea197f2bd21fe8379 WatchSource:0}: Error finding container 5cd7cc152fcf91331ae5b14f4e2e0c5c454c43037d53dd4ea197f2bd21fe8379: Status 404 returned error can't find the container with id 5cd7cc152fcf91331ae5b14f4e2e0c5c454c43037d53dd4ea197f2bd21fe8379 Nov 02 19:53:42.667606 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:42.667566 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" event={"ID":"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff","Type":"ContainerStarted","Data":"5cd7cc152fcf91331ae5b14f4e2e0c5c454c43037d53dd4ea197f2bd21fe8379"} Nov 02 19:53:45.683225 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:45.683128 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" event={"ID":"ba3a6e88-66b2-4b55-a673-6260a7cbbe1a","Type":"ContainerStarted","Data":"3d7e8842f50c47ea8330b98f2466385cd338a5cc4134edf3c681738c2114e1f3"} Nov 02 19:53:45.683662 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:45.683338 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:53:45.802994 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:45.802928 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" podStartSLOduration=1.424172262 podStartE2EDuration="5.802909607s" totalImagesPullingTime="4.378737345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:45.802319268 +0000 UTC m=+309.408863768" watchObservedRunningTime="2026-11-02 19:53:45.802909607 +0000 UTC m=+309.409454110" Nov 02 19:53:48.696915 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:48.696876 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" event={"ID":"0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff","Type":"ContainerStarted","Data":"c8594c1ce1f17f117e5fef5dce2b9196440db28e787c0e9af3acefb1cbb78b55"} Nov 02 19:53:48.807713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:48.807666 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pppkd" podStartSLOduration=2.7492502290000003 podStartE2EDuration="8.807654289s" totalImagesPullingTime="6.05840406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:48.807337716 +0000 UTC m=+312.413882215" watchObservedRunningTime="2026-11-02 19:53:48.807654289 +0000 UTC m=+312.414198786" Nov 02 19:53:49.831073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.831041 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n"] Nov 02 19:53:49.834635 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.834618 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:49.884453 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.884423 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 19:53:49.932161 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.932136 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 19:53:49.932161 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.932150 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mqm8f\"" Nov 02 19:53:49.933040 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.933018 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n"] Nov 02 19:53:49.960105 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.960080 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksmp\" (UniqueName: \"kubernetes.io/projected/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-kube-api-access-hksmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:49.960217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.960202 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:49.960268 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:49.960228 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.060975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.060951 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.060975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.060976 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.061127 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.061001 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksmp\" (UniqueName: \"kubernetes.io/projected/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-kube-api-access-hksmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.063245 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.063226 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.063371 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.063354 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-tls-certs\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.119027 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.118966 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksmp\" (UniqueName: \"kubernetes.io/projected/46cd203e-7ee4-4aa9-8b62-c5de1807bb2c-kube-api-access-hksmp\") pod \"kube-auth-proxy-7c9c5d7d8f-bs77n\" (UID: \"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c\") " pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.146814 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.146789 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" Nov 02 19:53:50.510227 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.510205 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n"] Nov 02 19:53:50.512237 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:53:50.512209 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd203e_7ee4_4aa9_8b62_c5de1807bb2c.slice/crio-a3ca33315b756e457139679de97b6a599cfe5f5691e043b9eadcdc42a49561f2 WatchSource:0}: Error finding container a3ca33315b756e457139679de97b6a599cfe5f5691e043b9eadcdc42a49561f2: Status 404 returned error can't find the container with id a3ca33315b756e457139679de97b6a599cfe5f5691e043b9eadcdc42a49561f2 Nov 02 19:53:50.705478 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:50.705442 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" event={"ID":"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c","Type":"ContainerStarted","Data":"a3ca33315b756e457139679de97b6a599cfe5f5691e043b9eadcdc42a49561f2"} Nov 02 19:53:51.002392 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:53:51.002358 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:53:52.248885 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.248854 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:53:52.252292 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.252269 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.305040 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.305016 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 19:53:52.306445 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.306417 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:53:52.381395 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.381360 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.381558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.381465 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6xs\" (UniqueName: \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.482531 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.482502 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.482703 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.482596 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6xs\" (UniqueName: \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.483293 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.483264 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.501326 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.501264 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:53:52.539163 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.539140 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6xs\" (UniqueName: \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\") pod \"limitador-limitador-74dfcfb894-6k4md\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.567479 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.567455 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:53:52.941842 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:52.941701 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:53:52.944070 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:53:52.944042 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb846c366_5bfe_4a0c_8f22_c97f74110ff8.slice/crio-75acb84439d84ac91fef54466c1c40b185e6d84aed932498e0aa4d531d975dd9 WatchSource:0}: Error finding container 75acb84439d84ac91fef54466c1c40b185e6d84aed932498e0aa4d531d975dd9: Status 404 returned error can't find the container with id 75acb84439d84ac91fef54466c1c40b185e6d84aed932498e0aa4d531d975dd9 Nov 02 19:53:53.718666 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:53.718629 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" event={"ID":"b846c366-5bfe-4a0c-8f22-c97f74110ff8","Type":"ContainerStarted","Data":"75acb84439d84ac91fef54466c1c40b185e6d84aed932498e0aa4d531d975dd9"} Nov 02 19:53:54.723450 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:54.723411 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" event={"ID":"46cd203e-7ee4-4aa9-8b62-c5de1807bb2c","Type":"ContainerStarted","Data":"2100c82dcb3053ff400d87cd0596a971ae95d41b3c8cfaa72f1b3aecacb95ff5"} Nov 02 19:53:54.831226 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:54.831178 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c9c5d7d8f-bs77n" podStartSLOduration=2.613296935 podStartE2EDuration="5.831165485s" totalImagesPullingTime="3.21786855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:54.830421367 +0000 UTC m=+318.436965867" watchObservedRunningTime="2026-11-02 19:53:54.831165485 +0000 UTC m=+318.437709980" Nov 02 19:53:56.690103 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:53:56.690072 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7cwmg" Nov 02 19:54:00.746745 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:00.746690 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" event={"ID":"b846c366-5bfe-4a0c-8f22-c97f74110ff8","Type":"ContainerStarted","Data":"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e"} Nov 02 19:54:00.747139 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:00.746872 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:54:00.881084 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:00.881033 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" podStartSLOduration=2.056915969 podStartE2EDuration="8.881021088s" totalImagesPullingTime="6.824105119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:00.879060057 +0000 UTC m=+324.485604554" watchObservedRunningTime="2026-11-02 19:54:00.881021088 +0000 UTC m=+324.487565598" Nov 02 19:54:11.753315 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:11.753284 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:54:11.799961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:11.799933 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:54:11.800155 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:11.800102 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" podUID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" containerName="limitador" containerID="cri-o://03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e" gracePeriod=30 Nov 02 19:54:12.334643 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.334617 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:54:12.435014 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.434942 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\" (UniqueName: \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\") pod \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " Nov 02 19:54:12.435168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.435033 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\" (UniqueName: \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\") pod \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\" (UID: \"b846c366-5bfe-4a0c-8f22-c97f74110ff8\") " Nov 02 19:54:12.435397 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.435375 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file" pod "b846c366-5bfe-4a0c-8f22-c97f74110ff8" (UID: "b846c366-5bfe-4a0c-8f22-c97f74110ff8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:12.437257 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.437230 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs" pod "b846c366-5bfe-4a0c-8f22-c97f74110ff8" (UID: "b846c366-5bfe-4a0c-8f22-c97f74110ff8"). InnerVolumeSpecName "kube-api-access-td6xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:12.535708 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.535673 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6xs\" (UniqueName: \"kubernetes.io/projected/b846c366-5bfe-4a0c-8f22-c97f74110ff8-kube-api-access-td6xs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:12.535708 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.535704 5136 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b846c366-5bfe-4a0c-8f22-c97f74110ff8-config-file\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:12.794604 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.794573 5136 generic.go:356] "Generic (PLEG): container finished" podID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" containerID="03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e" exitCode=0 Nov 02 19:54:12.795091 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.794619 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" Nov 02 19:54:12.795091 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.794668 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" event={"ID":"b846c366-5bfe-4a0c-8f22-c97f74110ff8","Type":"ContainerDied","Data":"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e"} Nov 02 19:54:12.795091 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.794720 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6k4md" event={"ID":"b846c366-5bfe-4a0c-8f22-c97f74110ff8","Type":"ContainerDied","Data":"75acb84439d84ac91fef54466c1c40b185e6d84aed932498e0aa4d531d975dd9"} Nov 02 19:54:12.795091 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.794759 5136 scope.go:122] "RemoveContainer" containerID="03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e" Nov 02 19:54:12.802873 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.802688 5136 scope.go:122] "RemoveContainer" containerID="03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e" Nov 02 19:54:12.802975 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:54:12.802957 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e\": container with ID starting with 03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e not found: ID does not exist" containerID="03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e" Nov 02 19:54:12.803017 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.802982 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e"} err="failed to get container status \"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e\": rpc error: code = NotFound desc = could not find container \"03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e\": container with ID starting with 03fb787836c7f2efeee373034507027735fff83be1c436ebae2834d09e39c74e not found: ID does not exist" Nov 02 19:54:12.950428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.950391 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:54:12.997357 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:12.997328 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6k4md"] Nov 02 19:54:14.544159 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:14.544121 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" path="/var/lib/kubelet/pods/b846c366-5bfe-4a0c-8f22-c97f74110ff8/volumes" Nov 02 19:54:15.124185 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.124152 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc"] Nov 02 19:54:15.124572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.124554 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" containerName="limitador" Nov 02 19:54:15.124612 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.124574 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" containerName="limitador" Nov 02 19:54:15.124645 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.124634 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b846c366-5bfe-4a0c-8f22-c97f74110ff8" containerName="limitador" Nov 02 19:54:15.128778 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.128756 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.156147 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.156116 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkp8\" (UniqueName: \"kubernetes.io/projected/95efd56e-e640-4249-b8f4-5e55d11f91c2-kube-api-access-7zkp8\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.156281 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.156169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.156328 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.156309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95efd56e-e640-4249-b8f4-5e55d11f91c2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.156377 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.156346 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.178499 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.178471 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 19:54:15.225744 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.225699 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 19:54:15.225923 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.225899 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-627fg\"" Nov 02 19:54:15.257385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257352 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkp8\" (UniqueName: \"kubernetes.io/projected/95efd56e-e640-4249-b8f4-5e55d11f91c2-kube-api-access-7zkp8\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.257515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257398 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.257515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257447 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95efd56e-e640-4249-b8f4-5e55d11f91c2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.257515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257466 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.257895 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257875 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.257983 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.257904 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/95efd56e-e640-4249-b8f4-5e55d11f91c2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.321668 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.321640 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 19:54:15.322328 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.322304 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc"] Nov 02 19:54:15.324072 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.324050 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkp8\" (UniqueName: \"kubernetes.io/projected/95efd56e-e640-4249-b8f4-5e55d11f91c2-kube-api-access-7zkp8\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.328435 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.328415 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95efd56e-e640-4249-b8f4-5e55d11f91c2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mg6sc\" (UID: \"95efd56e-e640-4249-b8f4-5e55d11f91c2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.440880 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.440811 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:15.850489 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:15.850466 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc"] Nov 02 19:54:15.853036 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:54:15.853007 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95efd56e_e640_4249_b8f4_5e55d11f91c2.slice/crio-a32c22e9e19920346936d252072e48cdf03261cae2d92178e7df8d936bd910a3 WatchSource:0}: Error finding container a32c22e9e19920346936d252072e48cdf03261cae2d92178e7df8d936bd910a3: Status 404 returned error can't find the container with id a32c22e9e19920346936d252072e48cdf03261cae2d92178e7df8d936bd910a3 Nov 02 19:54:16.812267 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:16.812225 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" event={"ID":"95efd56e-e640-4249-b8f4-5e55d11f91c2","Type":"ContainerStarted","Data":"a32c22e9e19920346936d252072e48cdf03261cae2d92178e7df8d936bd910a3"} Nov 02 19:54:18.823082 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:18.823044 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" event={"ID":"95efd56e-e640-4249-b8f4-5e55d11f91c2","Type":"ContainerStarted","Data":"4ece0d0da0407d64d47dffed262147667b547e521f2aa6e5868a73995a9b6596"} Nov 02 19:54:18.823483 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:18.823170 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:18.979857 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:18.979806 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" podStartSLOduration=1.919744969 podStartE2EDuration="3.979792808s" totalImagesPullingTime="2.060047839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:18.978985522 +0000 UTC m=+342.585530019" watchObservedRunningTime="2026-11-02 19:54:18.979792808 +0000 UTC m=+342.586337343" Nov 02 19:54:21.002443 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:54:21.002402 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:54:21.097313 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.097280 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq"] Nov 02 19:54:21.231089 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.231050 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq"] Nov 02 19:54:21.231242 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.231166 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.280981 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.280905 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 19:54:21.280981 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.280925 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bwmtb\"" Nov 02 19:54:21.313216 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.313187 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcppk\" (UniqueName: \"kubernetes.io/projected/1a9fc457-ac3f-4440-b120-10b514edc6a5-kube-api-access-fcppk\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.313353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.313245 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9fc457-ac3f-4440-b120-10b514edc6a5-data\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.328467 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.328441 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:54:21.328559 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.328500 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:54:21.414222 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.414186 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9fc457-ac3f-4440-b120-10b514edc6a5-data\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.414385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.414263 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcppk\" (UniqueName: \"kubernetes.io/projected/1a9fc457-ac3f-4440-b120-10b514edc6a5-kube-api-access-fcppk\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.414570 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.414552 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9fc457-ac3f-4440-b120-10b514edc6a5-data\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.468868 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.468833 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcppk\" (UniqueName: \"kubernetes.io/projected/1a9fc457-ac3f-4440-b120-10b514edc6a5-kube-api-access-fcppk\") pod \"postgres-7cdbf6dc44-wttbq\" (UID: \"1a9fc457-ac3f-4440-b120-10b514edc6a5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.568427 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.568337 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:21.958344 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:21.958316 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq"] Nov 02 19:54:21.961018 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:54:21.960988 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9fc457_ac3f_4440_b120_10b514edc6a5.slice/crio-e9bd27e93b114329a86f651d169e32e06f1c10ff2156932872444f882d50a855 WatchSource:0}: Error finding container e9bd27e93b114329a86f651d169e32e06f1c10ff2156932872444f882d50a855: Status 404 returned error can't find the container with id e9bd27e93b114329a86f651d169e32e06f1c10ff2156932872444f882d50a855 Nov 02 19:54:22.841658 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:22.841622 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" event={"ID":"1a9fc457-ac3f-4440-b120-10b514edc6a5","Type":"ContainerStarted","Data":"e9bd27e93b114329a86f651d169e32e06f1c10ff2156932872444f882d50a855"} Nov 02 19:54:28.869628 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:28.869588 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" event={"ID":"1a9fc457-ac3f-4440-b120-10b514edc6a5","Type":"ContainerStarted","Data":"9a14cab06951e72180e1eb577c7689a39aeaa80ad17296f0e7a5a4ccbb14bbab"} Nov 02 19:54:28.870143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:28.869705 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:28.977947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:28.977889 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" podStartSLOduration=1.522156896 podStartE2EDuration="7.977873533s" totalImagesPullingTime="6.455716637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:28.977615105 +0000 UTC m=+352.584159602" watchObservedRunningTime="2026-11-02 19:54:28.977873533 +0000 UTC m=+352.584418031" Nov 02 19:54:34.903200 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:34.903168 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wttbq" Nov 02 19:54:39.828261 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:39.828229 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mg6sc" Nov 02 19:54:41.712493 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.712461 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:54:41.718050 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.718028 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.768069 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.768035 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 19:54:41.786484 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.786452 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjmb\" (UniqueName: \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.786582 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.786488 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.786627 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.786599 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.789258 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.789238 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lfcl7\"" Nov 02 19:54:41.815331 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.815302 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 19:54:41.815504 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.815495 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 19:54:41.816506 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.816486 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:54:41.887146 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.887113 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjmb\" (UniqueName: \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.887146 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.887146 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.887348 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.887193 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.889612 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.889580 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.889725 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.889615 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:41.941290 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:41.941258 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjmb\" (UniqueName: \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\") pod \"maas-controller-68c8d8c8bb-mtxq8\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:42.030751 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.030631 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:42.272440 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.272407 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 19:54:42.277027 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.277005 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.329292 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.329262 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 19:54:42.393343 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.393316 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.393529 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.393350 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbmb\" (UniqueName: \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.393529 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.393377 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.477674 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.477649 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:54:42.479413 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:54:42.479388 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc586a_71ce_400c_a14c_296b334f4723.slice/crio-c7e0937cc9b0160084f02ca5d9d65e5438ff05065dabc7a05e471cbe9a1eeb71 WatchSource:0}: Error finding container c7e0937cc9b0160084f02ca5d9d65e5438ff05065dabc7a05e471cbe9a1eeb71: Status 404 returned error can't find the container with id c7e0937cc9b0160084f02ca5d9d65e5438ff05065dabc7a05e471cbe9a1eeb71 Nov 02 19:54:42.494707 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.494687 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbmb\" (UniqueName: \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.494799 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.494726 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.494845 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.494831 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.497084 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.497062 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.497161 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.497113 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.548866 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.548807 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbmb\" (UniqueName: \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\") pod \"maas-controller-94dcf6f94-snlvz\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.591707 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.591679 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:42.929827 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.929753 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" event={"ID":"40dc586a-71ce-400c-a14c-296b334f4723","Type":"ContainerStarted","Data":"c7e0937cc9b0160084f02ca5d9d65e5438ff05065dabc7a05e471cbe9a1eeb71"} Nov 02 19:54:42.954172 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:42.954152 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 19:54:42.956649 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:54:42.956608 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8be4d9a_6bc3_4c08_8ff0_c6bff5e3ca78.slice/crio-0cd09905b4a19dfab092f7cd52cc80e1a8e35f15c5a5f88f7d1ccfcda77a9b90 WatchSource:0}: Error finding container 0cd09905b4a19dfab092f7cd52cc80e1a8e35f15c5a5f88f7d1ccfcda77a9b90: Status 404 returned error can't find the container with id 0cd09905b4a19dfab092f7cd52cc80e1a8e35f15c5a5f88f7d1ccfcda77a9b90 Nov 02 19:54:43.934475 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:43.934439 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerStarted","Data":"0cd09905b4a19dfab092f7cd52cc80e1a8e35f15c5a5f88f7d1ccfcda77a9b90"} Nov 02 19:54:50.966720 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:50.966679 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerStarted","Data":"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af"} Nov 02 19:54:50.967175 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:50.966802 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:54:50.968380 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:50.968354 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" event={"ID":"40dc586a-71ce-400c-a14c-296b334f4723","Type":"ContainerStarted","Data":"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf"} Nov 02 19:54:50.968515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:50.968486 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:54:51.004690 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:54:51.004651 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:54:51.077168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:51.077115 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-94dcf6f94-snlvz" podStartSLOduration=1.6761276330000001 podStartE2EDuration="9.077103356s" totalImagesPullingTime="7.400975723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:51.076784081 +0000 UTC m=+374.683328580" watchObservedRunningTime="2026-11-02 19:54:51.077103356 +0000 UTC m=+374.683647905" Nov 02 19:54:51.185917 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:51.185864 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" podStartSLOduration=2.314928319 podStartE2EDuration="10.185848361s" totalImagesPullingTime="7.870920042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:51.185452433 +0000 UTC m=+374.791996930" watchObservedRunningTime="2026-11-02 19:54:51.185848361 +0000 UTC m=+374.792392859" Nov 02 19:54:53.837420 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.837382 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq"] Nov 02 19:54:53.841122 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.841101 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:53.892079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.892047 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 19:54:53.892079 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.892048 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 19:54:53.893858 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.893834 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq"] Nov 02 19:54:53.939337 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.939305 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 19:54:53.939487 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.939344 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-bwpzk\"" Nov 02 19:54:53.994726 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.994698 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:53.994899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.994813 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pk2k\" (UniqueName: \"kubernetes.io/projected/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-kube-api-access-9pk2k\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:53.994899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.994860 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:53.994986 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:53.994902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.096390 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.096309 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2k\" (UniqueName: \"kubernetes.io/projected/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-kube-api-access-9pk2k\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.096390 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.096370 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.096607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.096419 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.096607 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.096480 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.096607 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:54:54.096577 5136 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 19:54:54.096783 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:54:54.096678 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source podName:78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b nodeName:}" failed. No retries permitted until 2026-11-02 19:54:54.596654985 +0000 UTC m=+378.203199481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" (UID: "78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b") : configmap "usage-tenancy-proxy-source" not found Nov 02 19:54:54.097160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.097137 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.098927 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.098907 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.151635 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.151609 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pk2k\" (UniqueName: \"kubernetes.io/projected/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-kube-api-access-9pk2k\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.600845 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.600804 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.601596 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.601571 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-jb6hq\" (UID: \"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:54.753090 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:54.753048 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:54:55.115357 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:55.115328 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq"] Nov 02 19:54:55.117802 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:54:55.117712 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e9d3a6_8d7f_49d4_ab1f_9a408c4ae02b.slice/crio-de161fc28768009aade9ff4a284b747e6be399c7e163248068512b2ff2f439e3 WatchSource:0}: Error finding container de161fc28768009aade9ff4a284b747e6be399c7e163248068512b2ff2f439e3: Status 404 returned error can't find the container with id de161fc28768009aade9ff4a284b747e6be399c7e163248068512b2ff2f439e3 Nov 02 19:54:55.989680 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:54:55.989643 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" event={"ID":"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b","Type":"ContainerStarted","Data":"de161fc28768009aade9ff4a284b747e6be399c7e163248068512b2ff2f439e3"} Nov 02 19:55:01.975292 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:01.975262 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:55:01.975824 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:01.975339 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:55:02.134889 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.134251 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:55:02.134889 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.134475 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" podUID="40dc586a-71ce-400c-a14c-296b334f4723" containerName="manager" containerID="cri-o://0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf" gracePeriod=30 Nov 02 19:55:02.930671 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.930638 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:55:02.975169 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.975131 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\") pod \"40dc586a-71ce-400c-a14c-296b334f4723\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " Nov 02 19:55:02.975350 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.975207 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\" (UniqueName: \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\") pod \"40dc586a-71ce-400c-a14c-296b334f4723\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " Nov 02 19:55:02.975350 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.975302 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\") pod \"40dc586a-71ce-400c-a14c-296b334f4723\" (UID: \"40dc586a-71ce-400c-a14c-296b334f4723\") " Nov 02 19:55:02.980894 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.980861 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert" pod "40dc586a-71ce-400c-a14c-296b334f4723" (UID: "40dc586a-71ce-400c-a14c-296b334f4723"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:02.981002 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.980969 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb" pod "40dc586a-71ce-400c-a14c-296b334f4723" (UID: "40dc586a-71ce-400c-a14c-296b334f4723"). InnerVolumeSpecName "kube-api-access-rgjmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:02.981366 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:02.981343 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs" pod "40dc586a-71ce-400c-a14c-296b334f4723" (UID: "40dc586a-71ce-400c-a14c-296b334f4723"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:03.025487 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.025413 5136 generic.go:356] "Generic (PLEG): container finished" podID="40dc586a-71ce-400c-a14c-296b334f4723" containerID="0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf" exitCode=0 Nov 02 19:55:03.025487 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.025466 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" Nov 02 19:55:03.025695 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.025546 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" event={"ID":"40dc586a-71ce-400c-a14c-296b334f4723","Type":"ContainerDied","Data":"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf"} Nov 02 19:55:03.025695 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.025589 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-mtxq8" event={"ID":"40dc586a-71ce-400c-a14c-296b334f4723","Type":"ContainerDied","Data":"c7e0937cc9b0160084f02ca5d9d65e5438ff05065dabc7a05e471cbe9a1eeb71"} Nov 02 19:55:03.025695 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.025611 5136 scope.go:122] "RemoveContainer" containerID="0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf" Nov 02 19:55:03.045795 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.044774 5136 scope.go:122] "RemoveContainer" containerID="0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf" Nov 02 19:55:03.046343 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:55:03.046309 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf\": container with ID starting with 0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf not found: ID does not exist" containerID="0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf" Nov 02 19:55:03.046813 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.046356 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf"} err="failed to get container status \"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf\": rpc error: code = NotFound desc = could not find container \"0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf\": container with ID starting with 0208ebab5639716fd2adcb462aecbf5d69e2365042bcab690df20d4f6c326bdf not found: ID does not exist" Nov 02 19:55:03.076415 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.076382 5136 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-webhook-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:03.076557 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.076420 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc586a-71ce-400c-a14c-296b334f4723-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:03.076557 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.076439 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgjmb\" (UniqueName: \"kubernetes.io/projected/40dc586a-71ce-400c-a14c-296b334f4723-kube-api-access-rgjmb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:03.146514 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.146476 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:55:03.150525 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:03.150461 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-mtxq8"] Nov 02 19:55:04.547017 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:04.546978 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dc586a-71ce-400c-a14c-296b334f4723" path="/var/lib/kubelet/pods/40dc586a-71ce-400c-a14c-296b334f4723/volumes" Nov 02 19:55:09.057233 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:09.057191 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" event={"ID":"78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b","Type":"ContainerStarted","Data":"e49c6f16b9de55dab91e56a3bbeef627fd0c14ff742415e63a7f2f862c77e480"} Nov 02 19:55:09.057758 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:09.057345 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:55:09.169743 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:09.169682 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" podStartSLOduration=2.824554251 podStartE2EDuration="16.169669344s" totalImagesPullingTime="13.345115093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:09.168886792 +0000 UTC m=+392.775431289" watchObservedRunningTime="2026-11-02 19:55:09.169669344 +0000 UTC m=+392.776213841" Nov 02 19:55:15.168945 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.168915 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 19:55:15.169343 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.169277 5136 cpu_manager.go:414] "containerMap: removing container" podUID="40dc586a-71ce-400c-a14c-296b334f4723" containerName="manager" Nov 02 19:55:15.169343 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.169287 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dc586a-71ce-400c-a14c-296b334f4723" containerName="manager" Nov 02 19:55:15.169343 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.169343 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dc586a-71ce-400c-a14c-296b334f4723" containerName="manager" Nov 02 19:55:15.173724 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.173703 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.272388 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.272354 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 19:55:15.284663 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284628 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.284906 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284758 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.284906 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284817 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k5q\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.284906 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284848 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.284906 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284877 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.285076 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.284917 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.285115 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.285066 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385470 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385436 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385470 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385471 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385693 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385487 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385693 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385602 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385822 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385715 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385822 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385772 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.385822 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.385817 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k5q\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.386524 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.386497 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.387785 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.387765 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.387968 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.387948 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.388037 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.387954 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.388037 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.388021 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.439877 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.439816 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.440601 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.440577 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k5q\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbh62\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.484694 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.484670 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:15.894122 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.894097 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 19:55:15.895925 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:55:15.895892 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b7bbc7_a224_4e3b_bbf8_148479f943f4.slice/crio-04358a6d77db0ca4e07ab845450823881cffafb2d31352cf73b35124c185ee07 WatchSource:0}: Error finding container 04358a6d77db0ca4e07ab845450823881cffafb2d31352cf73b35124c185ee07: Status 404 returned error can't find the container with id 04358a6d77db0ca4e07ab845450823881cffafb2d31352cf73b35124c185ee07 Nov 02 19:55:15.898541 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.898505 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:15.898644 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:15.898581 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:16.087805 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.087764 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" event={"ID":"c8b7bbc7-a224-4e3b-bbf8-148479f943f4","Type":"ContainerStarted","Data":"418535d9375f2bb516f7119d93d6443159a8489197378f1b2d1414b2af62c4cf"} Nov 02 19:55:16.087991 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.087857 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" event={"ID":"c8b7bbc7-a224-4e3b-bbf8-148479f943f4","Type":"ContainerStarted","Data":"04358a6d77db0ca4e07ab845450823881cffafb2d31352cf73b35124c185ee07"} Nov 02 19:55:16.087991 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.087885 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:16.089288 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.089262 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lbh62 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:55:16.089394 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.089315 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:55:16.201002 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:16.200951 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" podStartSLOduration=1.20093773 podStartE2EDuration="1.20093773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:16.200790887 +0000 UTC m=+399.807335385" watchObservedRunningTime="2026-11-02 19:55:16.20093773 +0000 UTC m=+399.807482218" Nov 02 19:55:17.092637 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:17.092606 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lbh62 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:55:17.092812 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:17.092660 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:55:20.066989 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.066961 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-jb6hq" Nov 02 19:55:20.093103 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.093076 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 19:55:20.315795 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.315754 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:55:20.316092 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.316042 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" containerID="cri-o://616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9" gracePeriod=30 Nov 02 19:55:20.564485 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.564456 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:55:20.639063 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639025 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639063 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639078 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639109 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639157 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639214 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639278 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.639325 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.639307 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\") pod \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\" (UID: \"66ba7ff3-b4b2-4369-84dc-e59178de6a86\") " Nov 02 19:55:20.640410 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.640359 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:20.642573 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.642508 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:20.642758 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.642634 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:20.642758 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.642660 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:20.643052 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.643031 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:20.643131 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.643093 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "kube-api-access-pzt7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:20.643131 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.643097 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs" pod "66ba7ff3-b4b2-4369-84dc-e59178de6a86" (UID: "66ba7ff3-b4b2-4369-84dc-e59178de6a86"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:20.740796 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740759 5136 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba7ff3-b4b2-4369-84dc-e59178de6a86-local-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.740796 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740793 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.740796 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740803 5136 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-kubeconfig\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.741080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740812 5136 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-cacerts\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.741080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740823 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-ca-configmap\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.741080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740833 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzt7k\" (UniqueName: \"kubernetes.io/projected/66ba7ff3-b4b2-4369-84dc-e59178de6a86-kube-api-access-pzt7k\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:20.741080 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:20.740841 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66ba7ff3-b4b2-4369-84dc-e59178de6a86-istio-csr-dns-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:21.004024 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:55:21.003944 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:55:21.094899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.094859 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w9xl7"] Nov 02 19:55:21.095258 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.095230 5136 cpu_manager.go:414] "containerMap: removing container" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" Nov 02 19:55:21.095258 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.095242 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" Nov 02 19:55:21.095338 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.095312 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerName="discovery" Nov 02 19:55:21.098829 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.098807 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.111598 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.111570 5136 generic.go:356] "Generic (PLEG): container finished" podID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" containerID="616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9" exitCode=0 Nov 02 19:55:21.111713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.111613 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" Nov 02 19:55:21.111713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.111650 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" event={"ID":"66ba7ff3-b4b2-4369-84dc-e59178de6a86","Type":"ContainerDied","Data":"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9"} Nov 02 19:55:21.111713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.111696 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf" event={"ID":"66ba7ff3-b4b2-4369-84dc-e59178de6a86","Type":"ContainerDied","Data":"7dd345cc5ecc5ad61fafd7432583343a107c87bcb9250cfad4b823fa5013d185"} Nov 02 19:55:21.111850 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.111717 5136 scope.go:122] "RemoveContainer" containerID="616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9" Nov 02 19:55:21.122711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.122617 5136 scope.go:122] "RemoveContainer" containerID="616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9" Nov 02 19:55:21.123028 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:55:21.122974 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9\": container with ID starting with 616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9 not found: ID does not exist" containerID="616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9" Nov 02 19:55:21.123028 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.123008 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9"} err="failed to get container status \"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9\": rpc error: code = NotFound desc = could not find container \"616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9\": container with ID starting with 616b2405a2e263b5397ceaf8b26e1417b6ea14f8c74c070253ca0c90737241f9 not found: ID does not exist" Nov 02 19:55:21.151073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.151041 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 19:55:21.151073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.151041 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9qs7m\"" Nov 02 19:55:21.152247 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.152226 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w9xl7"] Nov 02 19:55:21.199502 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.199470 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2slch"] Nov 02 19:55:21.204225 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.204203 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.245641 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.245608 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e480fc-41e8-405f-a466-2e08f9041414-tls-certs\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.245821 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.245645 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sf4s\" (UniqueName: \"kubernetes.io/projected/73e480fc-41e8-405f-a466-2e08f9041414-kube-api-access-6sf4s\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.254708 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.254640 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 19:55:21.257023 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.256999 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2slch"] Nov 02 19:55:21.273203 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.273183 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 19:55:21.273336 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.273202 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q77q8\"" Nov 02 19:55:21.346816 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.346783 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5w2\" (UniqueName: \"kubernetes.io/projected/186a3e83-bf4f-4cca-972d-2761964a7c63-kube-api-access-km5w2\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.346999 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.346830 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-metrics-certs\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.346999 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.346855 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e480fc-41e8-405f-a466-2e08f9041414-tls-certs\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.346999 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.346900 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sf4s\" (UniqueName: \"kubernetes.io/projected/73e480fc-41e8-405f-a466-2e08f9041414-kube-api-access-6sf4s\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.347123 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.347057 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-cert\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.349297 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.349280 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e480fc-41e8-405f-a466-2e08f9041414-tls-certs\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.400662 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.400629 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sf4s\" (UniqueName: \"kubernetes.io/projected/73e480fc-41e8-405f-a466-2e08f9041414-kube-api-access-6sf4s\") pod \"model-serving-api-75bd89fdfc-w9xl7\" (UID: \"73e480fc-41e8-405f-a466-2e08f9041414\") " pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.412181 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.412158 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:21.420429 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.420342 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:55:21.423831 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.423804 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9tscf"] Nov 02 19:55:21.447800 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.447766 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-cert\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.447982 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.447826 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5w2\" (UniqueName: \"kubernetes.io/projected/186a3e83-bf4f-4cca-972d-2761964a7c63-kube-api-access-km5w2\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.447982 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.447870 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-metrics-certs\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.450339 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.450310 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-cert\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.450447 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.450323 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186a3e83-bf4f-4cca-972d-2761964a7c63-metrics-certs\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.506123 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.506055 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5w2\" (UniqueName: \"kubernetes.io/projected/186a3e83-bf4f-4cca-972d-2761964a7c63-kube-api-access-km5w2\") pod \"odh-model-controller-769875db56-2slch\" (UID: \"186a3e83-bf4f-4cca-972d-2761964a7c63\") " pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.516902 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.516878 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:21.775488 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.775463 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w9xl7"] Nov 02 19:55:21.777969 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:55:21.777943 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e480fc_41e8_405f_a466_2e08f9041414.slice/crio-94f0066457b5d4d64cf7c5a34fb7d4aa1b596042dfa3e9011eddcb6cb94669cc WatchSource:0}: Error finding container 94f0066457b5d4d64cf7c5a34fb7d4aa1b596042dfa3e9011eddcb6cb94669cc: Status 404 returned error can't find the container with id 94f0066457b5d4d64cf7c5a34fb7d4aa1b596042dfa3e9011eddcb6cb94669cc Nov 02 19:55:21.879600 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:21.879576 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2slch"] Nov 02 19:55:21.881555 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:55:21.881525 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186a3e83_bf4f_4cca_972d_2761964a7c63.slice/crio-859825c98e03b5f258ebc1e1fbe2414bf01307b435d14ffc0fd415db314d067a WatchSource:0}: Error finding container 859825c98e03b5f258ebc1e1fbe2414bf01307b435d14ffc0fd415db314d067a: Status 404 returned error can't find the container with id 859825c98e03b5f258ebc1e1fbe2414bf01307b435d14ffc0fd415db314d067a Nov 02 19:55:22.118255 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:22.118174 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" event={"ID":"73e480fc-41e8-405f-a466-2e08f9041414","Type":"ContainerStarted","Data":"94f0066457b5d4d64cf7c5a34fb7d4aa1b596042dfa3e9011eddcb6cb94669cc"} Nov 02 19:55:22.120149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:22.120120 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2slch" event={"ID":"186a3e83-bf4f-4cca-972d-2761964a7c63","Type":"ContainerStarted","Data":"859825c98e03b5f258ebc1e1fbe2414bf01307b435d14ffc0fd415db314d067a"} Nov 02 19:55:22.545565 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:22.545530 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba7ff3-b4b2-4369-84dc-e59178de6a86" path="/var/lib/kubelet/pods/66ba7ff3-b4b2-4369-84dc-e59178de6a86/volumes" Nov 02 19:55:24.810600 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.810529 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:24.815665 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.815642 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.875417 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.875386 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.875580 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.875510 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwx5l\" (UniqueName: \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.875648 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.875575 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.875648 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.875629 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.913116 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.913084 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 19:55:24.913323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.913213 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:55:24.913323 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.913213 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nltm4\"" Nov 02 19:55:24.914551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.914528 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:24.977428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.976856 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.977428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.976934 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwx5l\" (UniqueName: \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.977428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.976967 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.977428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.977011 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.977704 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.977598 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.979689 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.979661 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:24.979820 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:24.979774 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:25.030922 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.030895 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwx5l\" (UniqueName: \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\") pod \"maas-api-777f545dbb-pgsh9\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:25.130787 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.130702 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:25.139165 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.139120 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" event={"ID":"73e480fc-41e8-405f-a466-2e08f9041414","Type":"ContainerStarted","Data":"444e3245c4990d4d61e07b1f835426013eaa3748424e5c0a5237a0c79d833a2f"} Nov 02 19:55:25.139367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.139228 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:25.249012 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.248960 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" podStartSLOduration=1.516724418 podStartE2EDuration="4.248945227s" totalImagesPullingTime="2.732220809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:25.248302092 +0000 UTC m=+408.854846589" watchObservedRunningTime="2026-11-02 19:55:25.248945227 +0000 UTC m=+408.855489727" Nov 02 19:55:25.546099 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:25.546078 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:25.548322 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:55:25.548285 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706c2a17_2fe1_4250_bf4a_1636f9259de5.slice/crio-e94bafe4bc2648554174881092ba148c461a68ee385cb8ad6f21917c91a7555d WatchSource:0}: Error finding container e94bafe4bc2648554174881092ba148c461a68ee385cb8ad6f21917c91a7555d: Status 404 returned error can't find the container with id e94bafe4bc2648554174881092ba148c461a68ee385cb8ad6f21917c91a7555d Nov 02 19:55:26.145436 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:26.145398 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2slch" event={"ID":"186a3e83-bf4f-4cca-972d-2761964a7c63","Type":"ContainerStarted","Data":"ed765f22574bc4f845e7dcc9078a26da5f5439a488aac65da2d5b3a5811a611a"} Nov 02 19:55:26.145923 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:26.145548 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:26.146506 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:26.146485 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" event={"ID":"706c2a17-2fe1-4250-bf4a-1636f9259de5","Type":"ContainerStarted","Data":"e94bafe4bc2648554174881092ba148c461a68ee385cb8ad6f21917c91a7555d"} Nov 02 19:55:26.255407 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:26.255353 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2slch" podStartSLOduration=1.688819259 podStartE2EDuration="5.255339609s" totalImagesPullingTime="3.56652035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:26.254339976 +0000 UTC m=+409.860884475" watchObservedRunningTime="2026-11-02 19:55:26.255339609 +0000 UTC m=+409.861884119" Nov 02 19:55:33.179860 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:33.179813 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" event={"ID":"706c2a17-2fe1-4250-bf4a-1636f9259de5","Type":"ContainerStarted","Data":"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906"} Nov 02 19:55:33.180349 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:33.180069 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:33.288459 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:33.288407 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" podStartSLOduration=2.200394701 podStartE2EDuration="9.288393868s" totalImagesPullingTime="7.087999167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:33.28786042 +0000 UTC m=+416.894404954" watchObservedRunningTime="2026-11-02 19:55:33.288393868 +0000 UTC m=+416.894938365" Nov 02 19:55:36.150470 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:36.150437 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-w9xl7" Nov 02 19:55:37.154011 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:37.153978 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2slch" Nov 02 19:55:39.190312 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:39.190282 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:40.159925 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.159895 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 19:55:40.444257 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.444167 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 19:55:40.444707 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.444297 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.610133 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.610096 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.610133 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.610134 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkl8m\" (UniqueName: \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.610434 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.610237 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.610434 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.610288 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.711460 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.711425 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.711609 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.711496 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.711609 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.711534 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkl8m\" (UniqueName: \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.711681 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.711657 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.712240 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.712215 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.714057 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.714035 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.714164 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.714116 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.766190 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.766164 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkl8m\" (UniqueName: \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\") pod \"maas-api-794d57d6f9-5v8sj\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:40.790058 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:40.790030 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:41.199853 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:41.199831 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 19:55:41.201676 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:55:41.201651 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188c464b_11a9_452f_8b45_de935f8b28fb.slice/crio-bfaaa626db14e97bc5fa4c85011c3cad657262ea77351aa70cf9d436b4779b68 WatchSource:0}: Error finding container bfaaa626db14e97bc5fa4c85011c3cad657262ea77351aa70cf9d436b4779b68: Status 404 returned error can't find the container with id bfaaa626db14e97bc5fa4c85011c3cad657262ea77351aa70cf9d436b4779b68 Nov 02 19:55:41.220218 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:41.220184 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" event={"ID":"188c464b-11a9-452f-8b45-de935f8b28fb","Type":"ContainerStarted","Data":"bfaaa626db14e97bc5fa4c85011c3cad657262ea77351aa70cf9d436b4779b68"} Nov 02 19:55:42.226451 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:42.226412 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" event={"ID":"188c464b-11a9-452f-8b45-de935f8b28fb","Type":"ContainerStarted","Data":"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f"} Nov 02 19:55:42.226894 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:42.226533 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:42.335763 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:42.335660 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" podStartSLOduration=1.4696127940000001 podStartE2EDuration="2.335648808s" totalImagesPullingTime="866.036014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:42.334594601 +0000 UTC m=+425.941139100" watchObservedRunningTime="2026-11-02 19:55:42.335648808 +0000 UTC m=+425.942193306" Nov 02 19:55:48.235781 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.235751 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 19:55:48.375537 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.375505 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:48.375770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.375718 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" podUID="706c2a17-2fe1-4250-bf4a-1636f9259de5" containerName="maas-api" containerID="cri-o://1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906" gracePeriod=30 Nov 02 19:55:48.618266 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.618242 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:48.782510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.782433 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\" (UniqueName: \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\") pod \"706c2a17-2fe1-4250-bf4a-1636f9259de5\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " Nov 02 19:55:48.782510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.782472 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\") pod \"706c2a17-2fe1-4250-bf4a-1636f9259de5\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " Nov 02 19:55:48.782510 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.782497 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\") pod \"706c2a17-2fe1-4250-bf4a-1636f9259de5\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " Nov 02 19:55:48.782827 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.782529 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\") pod \"706c2a17-2fe1-4250-bf4a-1636f9259de5\" (UID: \"706c2a17-2fe1-4250-bf4a-1636f9259de5\") " Nov 02 19:55:48.782962 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.782930 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca" pod "706c2a17-2fe1-4250-bf4a-1636f9259de5" (UID: "706c2a17-2fe1-4250-bf4a-1636f9259de5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:48.784863 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.784835 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls" pod "706c2a17-2fe1-4250-bf4a-1636f9259de5" (UID: "706c2a17-2fe1-4250-bf4a-1636f9259de5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:48.784948 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.784876 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs" pod "706c2a17-2fe1-4250-bf4a-1636f9259de5" (UID: "706c2a17-2fe1-4250-bf4a-1636f9259de5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:48.784948 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.784888 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l" pod "706c2a17-2fe1-4250-bf4a-1636f9259de5" (UID: "706c2a17-2fe1-4250-bf4a-1636f9259de5"). InnerVolumeSpecName "kube-api-access-dwx5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:48.884032 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.884000 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwx5l\" (UniqueName: \"kubernetes.io/projected/706c2a17-2fe1-4250-bf4a-1636f9259de5-kube-api-access-dwx5l\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:48.884032 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.884027 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:48.884209 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.884041 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c2a17-2fe1-4250-bf4a-1636f9259de5-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:48.884209 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:48.884054 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706c2a17-2fe1-4250-bf4a-1636f9259de5-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:49.256650 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.256612 5136 generic.go:356] "Generic (PLEG): container finished" podID="706c2a17-2fe1-4250-bf4a-1636f9259de5" containerID="1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906" exitCode=0 Nov 02 19:55:49.257081 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.256666 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" Nov 02 19:55:49.257081 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.256701 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" event={"ID":"706c2a17-2fe1-4250-bf4a-1636f9259de5","Type":"ContainerDied","Data":"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906"} Nov 02 19:55:49.257081 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.256775 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9" event={"ID":"706c2a17-2fe1-4250-bf4a-1636f9259de5","Type":"ContainerDied","Data":"e94bafe4bc2648554174881092ba148c461a68ee385cb8ad6f21917c91a7555d"} Nov 02 19:55:49.257081 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.256800 5136 scope.go:122] "RemoveContainer" containerID="1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906" Nov 02 19:55:49.266284 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.266265 5136 scope.go:122] "RemoveContainer" containerID="1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906" Nov 02 19:55:49.266543 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:55:49.266521 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906\": container with ID starting with 1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906 not found: ID does not exist" containerID="1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906" Nov 02 19:55:49.266602 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.266550 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906"} err="failed to get container status \"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906\": rpc error: code = NotFound desc = could not find container \"1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906\": container with ID starting with 1dfc47dc7b54585eb2f3f16f91c7a5662ca11f633dcfa74ae5c3f3afa3ead906 not found: ID does not exist" Nov 02 19:55:49.410234 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.410208 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:49.457562 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:49.457535 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-pgsh9"] Nov 02 19:55:50.545053 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:55:50.545020 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706c2a17-2fe1-4250-bf4a-1636f9259de5" path="/var/lib/kubelet/pods/706c2a17-2fe1-4250-bf4a-1636f9259de5/volumes" Nov 02 19:55:51.003550 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:55:51.003521 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:56:21.002552 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:56:21.002469 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:56:51.003535 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:56:51.003493 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:56:59.653602 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.653558 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:56:59.654179 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.653973 5136 cpu_manager.go:414] "containerMap: removing container" podUID="706c2a17-2fe1-4250-bf4a-1636f9259de5" containerName="maas-api" Nov 02 19:56:59.654179 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.653986 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c2a17-2fe1-4250-bf4a-1636f9259de5" containerName="maas-api" Nov 02 19:56:59.654179 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.654065 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c2a17-2fe1-4250-bf4a-1636f9259de5" containerName="maas-api" Nov 02 19:56:59.656261 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.656246 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.705959 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.705930 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 19:56:59.753356 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.753319 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 19:56:59.753356 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.753346 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-8qq4k\"" Nov 02 19:56:59.754623 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.754594 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:56:59.772520 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.772491 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8r9f\" (UniqueName: \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.772628 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.772564 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.873357 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.873325 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.873523 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.873380 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8r9f\" (UniqueName: \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.876056 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.876028 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.927055 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.926986 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8r9f\" (UniqueName: \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\") pod \"ai-gateway-controller-5d8dddb676-n2dml\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:56:59.968044 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:56:59.968013 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:57:00.330214 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:00.330192 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:57:00.332292 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:57:00.332265 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b348fe2_c1ea_4824_951f_0676591cc2e8.slice/crio-dcc5a5f51b369e3d610f3d01ec3524881b3fe69be6fdc4ef8541e4a75ccf6b40 WatchSource:0}: Error finding container dcc5a5f51b369e3d610f3d01ec3524881b3fe69be6fdc4ef8541e4a75ccf6b40: Status 404 returned error can't find the container with id dcc5a5f51b369e3d610f3d01ec3524881b3fe69be6fdc4ef8541e4a75ccf6b40 Nov 02 19:57:00.560358 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:00.560327 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" event={"ID":"6b348fe2-c1ea-4824-951f-0676591cc2e8","Type":"ContainerStarted","Data":"dcc5a5f51b369e3d610f3d01ec3524881b3fe69be6fdc4ef8541e4a75ccf6b40"} Nov 02 19:57:05.596088 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:05.596050 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" event={"ID":"6b348fe2-c1ea-4824-951f-0676591cc2e8","Type":"ContainerStarted","Data":"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0"} Nov 02 19:57:05.596513 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:05.596157 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:57:05.705963 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:05.705917 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" podStartSLOduration=1.851914091 podStartE2EDuration="6.705902173s" totalImagesPullingTime="4.853988082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:05.704911683 +0000 UTC m=+509.311456181" watchObservedRunningTime="2026-11-02 19:57:05.705902173 +0000 UTC m=+509.312446664" Nov 02 19:57:09.536368 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.536309 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 19:57:09.538973 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.538955 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.622859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.622829 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-msppk\"" Nov 02 19:57:09.630182 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.630162 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 19:57:09.631271 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.631251 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 19:57:09.665560 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.665529 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxk67\" (UniqueName: \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.665680 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.665597 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.767072 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.767033 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.767223 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.767126 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxk67\" (UniqueName: \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.767747 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.767713 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.822140 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.822074 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxk67\" (UniqueName: \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\") pod \"payload-processing-55fbbbf9dc-vzlpf\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:09.850706 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:09.850679 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:10.216941 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:10.216919 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 19:57:10.218597 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:57:10.218572 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe75c77e_1b1d_4489_9a57_d8d6d1fffc7f.slice/crio-4be1bb8cc2fcb85c0afd633102721d8424dc0cab54ba08e2927a2b60aa10a1ed WatchSource:0}: Error finding container 4be1bb8cc2fcb85c0afd633102721d8424dc0cab54ba08e2927a2b60aa10a1ed: Status 404 returned error can't find the container with id 4be1bb8cc2fcb85c0afd633102721d8424dc0cab54ba08e2927a2b60aa10a1ed Nov 02 19:57:10.618794 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:10.618686 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" event={"ID":"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f","Type":"ContainerStarted","Data":"4be1bb8cc2fcb85c0afd633102721d8424dc0cab54ba08e2927a2b60aa10a1ed"} Nov 02 19:57:13.635252 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:13.635216 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" event={"ID":"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f","Type":"ContainerStarted","Data":"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd"} Nov 02 19:57:13.635758 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:13.635278 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:13.743206 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:13.743160 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" podStartSLOduration=2.27396843 podStartE2EDuration="4.743147573s" totalImagesPullingTime="2.469179143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:13.742480583 +0000 UTC m=+517.349025081" watchObservedRunningTime="2026-11-02 19:57:13.743147573 +0000 UTC m=+517.349692065" Nov 02 19:57:16.601830 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:16.601802 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:57:16.933949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:16.933875 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-656cc65bfd-xlcwc"] Nov 02 19:57:16.936575 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:16.936559 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:16.991432 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:16.991393 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-656cc65bfd-xlcwc"] Nov 02 19:57:17.033004 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.032974 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78556d8e-59a8-4977-8d09-8586ee3cb6c7-metrics-certs\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.033143 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.033094 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrz7\" (UniqueName: \"kubernetes.io/projected/78556d8e-59a8-4977-8d09-8586ee3cb6c7-kube-api-access-gnrz7\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.134116 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.134085 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78556d8e-59a8-4977-8d09-8586ee3cb6c7-metrics-certs\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.134260 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.134181 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrz7\" (UniqueName: \"kubernetes.io/projected/78556d8e-59a8-4977-8d09-8586ee3cb6c7-kube-api-access-gnrz7\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.136523 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.136502 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78556d8e-59a8-4977-8d09-8586ee3cb6c7-metrics-certs\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.189590 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.189528 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrz7\" (UniqueName: \"kubernetes.io/projected/78556d8e-59a8-4977-8d09-8586ee3cb6c7-kube-api-access-gnrz7\") pod \"ai-gateway-controller-656cc65bfd-xlcwc\" (UID: \"78556d8e-59a8-4977-8d09-8586ee3cb6c7\") " pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.255414 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.255383 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:17.615780 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.615757 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-656cc65bfd-xlcwc"] Nov 02 19:57:17.617857 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:57:17.617828 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78556d8e_59a8_4977_8d09_8586ee3cb6c7.slice/crio-015c12cf559b5ab4c686f1ea2c8c2b85c69fa9929e0f8eca10fde8c9cc7400d0 WatchSource:0}: Error finding container 015c12cf559b5ab4c686f1ea2c8c2b85c69fa9929e0f8eca10fde8c9cc7400d0: Status 404 returned error can't find the container with id 015c12cf559b5ab4c686f1ea2c8c2b85c69fa9929e0f8eca10fde8c9cc7400d0 Nov 02 19:57:17.652859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:17.652834 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" event={"ID":"78556d8e-59a8-4977-8d09-8586ee3cb6c7","Type":"ContainerStarted","Data":"015c12cf559b5ab4c686f1ea2c8c2b85c69fa9929e0f8eca10fde8c9cc7400d0"} Nov 02 19:57:18.659007 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:18.658968 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" event={"ID":"78556d8e-59a8-4977-8d09-8586ee3cb6c7","Type":"ContainerStarted","Data":"cf7641883a8b6df622f6c63095196fee4827c355a85a2b4564a19b525ff0381a"} Nov 02 19:57:18.659415 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:18.659117 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:18.770153 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:18.770057 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" podStartSLOduration=1.937191115 podStartE2EDuration="2.770043968s" totalImagesPullingTime="832.852853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:18.768783182 +0000 UTC m=+522.375327679" watchObservedRunningTime="2026-11-02 19:57:18.770043968 +0000 UTC m=+522.376588465" Nov 02 19:57:19.664433 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:19.664354 5136 generic.go:356] "Generic (PLEG): container finished" podID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerID="70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af" exitCode=1 Nov 02 19:57:19.664853 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:19.664434 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerDied","Data":"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af"} Nov 02 19:57:19.664899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:19.664869 5136 scope.go:122] "RemoveContainer" containerID="70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af" Nov 02 19:57:21.004970 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:57:21.004939 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:57:21.675499 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:21.675464 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerStarted","Data":"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d"} Nov 02 19:57:21.675700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:21.675677 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:57:24.642173 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:24.642141 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 19:57:29.666920 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:29.666891 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-656cc65bfd-xlcwc" Nov 02 19:57:29.824958 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:29.824925 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:57:29.825192 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:29.825168 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" podUID="6b348fe2-c1ea-4824-951f-0676591cc2e8" containerName="manager" containerID="cri-o://6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0" gracePeriod=30 Nov 02 19:57:30.070353 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.070327 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:57:30.149674 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.149642 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\" (UniqueName: \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\") pod \"6b348fe2-c1ea-4824-951f-0676591cc2e8\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " Nov 02 19:57:30.149674 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.149672 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\") pod \"6b348fe2-c1ea-4824-951f-0676591cc2e8\" (UID: \"6b348fe2-c1ea-4824-951f-0676591cc2e8\") " Nov 02 19:57:30.151705 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.151674 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f" pod "6b348fe2-c1ea-4824-951f-0676591cc2e8" (UID: "6b348fe2-c1ea-4824-951f-0676591cc2e8"). InnerVolumeSpecName "kube-api-access-w8r9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:30.151850 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.151708 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs" pod "6b348fe2-c1ea-4824-951f-0676591cc2e8" (UID: "6b348fe2-c1ea-4824-951f-0676591cc2e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:57:30.250965 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.250934 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8r9f\" (UniqueName: \"kubernetes.io/projected/6b348fe2-c1ea-4824-951f-0676591cc2e8-kube-api-access-w8r9f\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:30.250965 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.250961 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b348fe2-c1ea-4824-951f-0676591cc2e8-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:30.717865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.717830 5136 generic.go:356] "Generic (PLEG): container finished" podID="6b348fe2-c1ea-4824-951f-0676591cc2e8" containerID="6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0" exitCode=0 Nov 02 19:57:30.718314 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.717879 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" Nov 02 19:57:30.718314 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.717918 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" event={"ID":"6b348fe2-c1ea-4824-951f-0676591cc2e8","Type":"ContainerDied","Data":"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0"} Nov 02 19:57:30.718314 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.717971 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-n2dml" event={"ID":"6b348fe2-c1ea-4824-951f-0676591cc2e8","Type":"ContainerDied","Data":"dcc5a5f51b369e3d610f3d01ec3524881b3fe69be6fdc4ef8541e4a75ccf6b40"} Nov 02 19:57:30.718314 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.717993 5136 scope.go:122] "RemoveContainer" containerID="6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0" Nov 02 19:57:30.726111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.726093 5136 scope.go:122] "RemoveContainer" containerID="6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0" Nov 02 19:57:30.726382 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:57:30.726353 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0\": container with ID starting with 6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0 not found: ID does not exist" containerID="6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0" Nov 02 19:57:30.726433 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.726390 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0"} err="failed to get container status \"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0\": rpc error: code = NotFound desc = could not find container \"6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0\": container with ID starting with 6f8d6d3c26f9e762a73ba88142a0a4cfd0e8df343ff8560cfce83626890749d0 not found: ID does not exist" Nov 02 19:57:30.872404 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.872334 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:57:30.879030 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:30.879006 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-n2dml"] Nov 02 19:57:32.544859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:32.544826 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b348fe2-c1ea-4824-951f-0676591cc2e8" path="/var/lib/kubelet/pods/6b348fe2-c1ea-4824-951f-0676591cc2e8/volumes" Nov 02 19:57:32.685099 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:32.685074 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 19:57:40.477390 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.477354 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc"] Nov 02 19:57:40.477813 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.477787 5136 cpu_manager.go:414] "containerMap: removing container" podUID="6b348fe2-c1ea-4824-951f-0676591cc2e8" containerName="manager" Nov 02 19:57:40.477813 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.477815 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b348fe2-c1ea-4824-951f-0676591cc2e8" containerName="manager" Nov 02 19:57:40.477904 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.477886 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b348fe2-c1ea-4824-951f-0676591cc2e8" containerName="manager" Nov 02 19:57:40.483870 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.483848 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.533763 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.533713 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.533941 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.533811 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.533941 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.533903 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.534073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.533962 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmbz\" (UniqueName: \"kubernetes.io/projected/604a1fb3-761d-442c-8ef7-076a37af476d-kube-api-access-qkmbz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.534073 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534006 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.534183 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534078 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 19:57:40.534183 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534105 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 19:57:40.534278 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534192 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604a1fb3-761d-442c-8ef7-076a37af476d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.534278 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534203 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 19:57:40.534278 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534208 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7j5x\"" Nov 02 19:57:40.534409 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.534352 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc"] Nov 02 19:57:40.634965 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.634921 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635159 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.634988 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635159 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635017 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmbz\" (UniqueName: \"kubernetes.io/projected/604a1fb3-761d-442c-8ef7-076a37af476d-kube-api-access-qkmbz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635159 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635047 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635159 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635148 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604a1fb3-761d-442c-8ef7-076a37af476d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635384 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635188 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635447 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635383 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635447 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635410 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.635662 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.635638 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.637440 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.637419 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604a1fb3-761d-442c-8ef7-076a37af476d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.637792 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.637773 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604a1fb3-761d-442c-8ef7-076a37af476d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.691509 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.691476 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmbz\" (UniqueName: \"kubernetes.io/projected/604a1fb3-761d-442c-8ef7-076a37af476d-kube-api-access-qkmbz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc\" (UID: \"604a1fb3-761d-442c-8ef7-076a37af476d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:40.797196 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:40.797103 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:41.161615 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:41.161594 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc"] Nov 02 19:57:41.163159 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:57:41.163134 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604a1fb3_761d_442c_8ef7_076a37af476d.slice/crio-20212a8578035d9e582a011040056f61a2191f37728385972d59d10eb0468e91 WatchSource:0}: Error finding container 20212a8578035d9e582a011040056f61a2191f37728385972d59d10eb0468e91: Status 404 returned error can't find the container with id 20212a8578035d9e582a011040056f61a2191f37728385972d59d10eb0468e91 Nov 02 19:57:41.766616 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:41.766581 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" event={"ID":"604a1fb3-761d-442c-8ef7-076a37af476d","Type":"ContainerStarted","Data":"20212a8578035d9e582a011040056f61a2191f37728385972d59d10eb0468e91"} Nov 02 19:57:43.776168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:43.776119 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" event={"ID":"604a1fb3-761d-442c-8ef7-076a37af476d","Type":"ContainerStarted","Data":"2041a1baa8dfa856ef1784e186aec9f49db1c00c3e0e1c988c1835f6480eb33d"} Nov 02 19:57:43.798044 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:43.798018 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:43.885850 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:43.885801 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" podStartSLOduration=1.758560291 podStartE2EDuration="3.885787718s" totalImagesPullingTime="2.127227427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:43.884761018 +0000 UTC m=+547.491305516" watchObservedRunningTime="2026-11-02 19:57:43.885787718 +0000 UTC m=+547.492332216" Nov 02 19:57:50.798218 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:50.798135 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:50.810930 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:50.810904 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:50.822176 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:50.822155 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc" Nov 02 19:57:51.002258 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:57:51.002225 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:57:55.941181 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:55.941147 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj"] Nov 02 19:57:55.944604 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:55.944585 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.043054 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.043027 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 19:57:56.044347 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.044327 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj"] Nov 02 19:57:56.070591 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070562 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.070707 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070598 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxn5\" (UniqueName: \"kubernetes.io/projected/46d6c4bb-3db2-4846-9a60-ada0cd07e351-kube-api-access-2wxn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.070707 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070633 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.070804 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070775 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.070859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070843 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.070893 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.070878 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172334 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172302 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172354 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172376 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxn5\" (UniqueName: \"kubernetes.io/projected/46d6c4bb-3db2-4846-9a60-ada0cd07e351-kube-api-access-2wxn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172398 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172418 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172467 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172817 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172780 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172865 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172845 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.172899 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.172864 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.174850 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.174823 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d6c4bb-3db2-4846-9a60-ada0cd07e351-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.175047 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.175032 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d6c4bb-3db2-4846-9a60-ada0cd07e351-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.229587 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.229555 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxn5\" (UniqueName: \"kubernetes.io/projected/46d6c4bb-3db2-4846-9a60-ada0cd07e351-kube-api-access-2wxn5\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z2zkj\" (UID: \"46d6c4bb-3db2-4846-9a60-ada0cd07e351\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.257126 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.257089 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:57:56.624354 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.624331 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj"] Nov 02 19:57:56.626402 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:57:56.626374 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d6c4bb_3db2_4846_9a60_ada0cd07e351.slice/crio-8d26f50ebaa4d5a7f3f98c914d8491215fc5912ea4158b3c812862521262a59b WatchSource:0}: Error finding container 8d26f50ebaa4d5a7f3f98c914d8491215fc5912ea4158b3c812862521262a59b: Status 404 returned error can't find the container with id 8d26f50ebaa4d5a7f3f98c914d8491215fc5912ea4158b3c812862521262a59b Nov 02 19:57:56.832496 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:56.832464 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" event={"ID":"46d6c4bb-3db2-4846-9a60-ada0cd07e351","Type":"ContainerStarted","Data":"8d26f50ebaa4d5a7f3f98c914d8491215fc5912ea4158b3c812862521262a59b"} Nov 02 19:57:57.838611 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:57.838566 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" event={"ID":"46d6c4bb-3db2-4846-9a60-ada0cd07e351","Type":"ContainerStarted","Data":"ea5524ba8708e97f99586117fcdc8809cec1cad978e2a29a3d0f942807e0d661"} Nov 02 19:57:57.947827 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:57.947782 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" podStartSLOduration=2.3347126400000002 podStartE2EDuration="2.947768019s" totalImagesPullingTime="613.055379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:57.947065148 +0000 UTC m=+561.553609650" watchObservedRunningTime="2026-11-02 19:57:57.947768019 +0000 UTC m=+561.554312516" Nov 02 19:57:58.257423 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:57:58.257392 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:58:03.309172 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.309135 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4lsbj"] Nov 02 19:58:03.312928 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.312901 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.411321 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.411284 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 19:58:03.412660 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.412639 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4lsbj"] Nov 02 19:58:03.437546 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437518 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.437704 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437569 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.437704 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437587 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.437862 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437751 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-home\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.437862 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437814 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55wh\" (UniqueName: \"kubernetes.io/projected/8ec662da-351c-4ed1-a35b-1a0fea0f294c-kube-api-access-n55wh\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.437934 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.437905 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539202 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539173 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539219 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539261 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539276 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539367 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539293 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-home\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539378 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55wh\" (UniqueName: \"kubernetes.io/projected/8ec662da-351c-4ed1-a35b-1a0fea0f294c-kube-api-access-n55wh\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539649 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539626 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539713 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539677 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-home\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.539798 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.539724 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.541595 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.541568 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec662da-351c-4ed1-a35b-1a0fea0f294c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.541818 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.541802 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec662da-351c-4ed1-a35b-1a0fea0f294c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.594030 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.593951 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55wh\" (UniqueName: \"kubernetes.io/projected/8ec662da-351c-4ed1-a35b-1a0fea0f294c-kube-api-access-n55wh\") pod \"echo-simulator-kserve-64b6f8f96c-4lsbj\" (UID: \"8ec662da-351c-4ed1-a35b-1a0fea0f294c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.625769 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.625723 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:03.994267 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:03.994237 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4lsbj"] Nov 02 19:58:03.995885 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:58:03.995858 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec662da_351c_4ed1_a35b_1a0fea0f294c.slice/crio-39d260697a56be796823e1f2d88698397e4ed27bcba62daa1b2d4902fc1d3d66 WatchSource:0}: Error finding container 39d260697a56be796823e1f2d88698397e4ed27bcba62daa1b2d4902fc1d3d66: Status 404 returned error can't find the container with id 39d260697a56be796823e1f2d88698397e4ed27bcba62daa1b2d4902fc1d3d66 Nov 02 19:58:04.870218 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:04.870187 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" event={"ID":"8ec662da-351c-4ed1-a35b-1a0fea0f294c","Type":"ContainerStarted","Data":"39d260697a56be796823e1f2d88698397e4ed27bcba62daa1b2d4902fc1d3d66"} Nov 02 19:58:06.257478 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.257437 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:58:06.272414 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.272388 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:58:06.881460 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.881369 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" event={"ID":"8ec662da-351c-4ed1-a35b-1a0fea0f294c","Type":"ContainerStarted","Data":"31e4ea4e9e934e78b225bc2ceec65dd72995ff760078a79f67bbc032b6a29949"} Nov 02 19:58:06.892964 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.892940 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z2zkj" Nov 02 19:58:06.904282 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.904261 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 19:58:06.907941 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.907925 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:06.992754 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.992707 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 19:58:06.995248 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:06.995202 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" podStartSLOduration=1.445670751 podStartE2EDuration="3.995187352s" totalImagesPullingTime="2.549516601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:06.993677001 +0000 UTC m=+570.600221490" watchObservedRunningTime="2026-11-02 19:58:06.995187352 +0000 UTC m=+570.601731849" Nov 02 19:58:07.045200 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.045172 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\"" Nov 02 19:58:07.070368 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070334 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070529 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070385 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070529 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070411 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070529 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070437 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070655 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070520 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070655 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070648 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070780 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070673 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070845 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070808 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.070897 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.070865 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98f9\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172270 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172183 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172270 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172219 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172511 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172346 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172511 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172372 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172511 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172418 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172511 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172464 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98f9\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172726 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172517 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172726 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172552 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.172726 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172578 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.173015 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.172954 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.173287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.173238 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.173287 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.173259 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.173445 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.173361 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.173592 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.173572 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.175028 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.175007 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.175494 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.175474 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.225989 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.225960 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.226126 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.226085 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98f9\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t8n2r\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.520700 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.520668 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:07.626870 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:07.626843 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:08.020344 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:08.020313 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 19:58:08.024841 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:58:08.024815 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8ffee0_608f_4c70_bbfc_6d1f34a53460.slice/crio-fedc24105f934e76b325a21713e154a00e668f283ff834371fd1929891a60c8a WatchSource:0}: Error finding container fedc24105f934e76b325a21713e154a00e668f283ff834371fd1929891a60c8a: Status 404 returned error can't find the container with id fedc24105f934e76b325a21713e154a00e668f283ff834371fd1929891a60c8a Nov 02 19:58:08.899770 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:08.899723 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" event={"ID":"7f8ffee0-608f-4c70-bbfc-6d1f34a53460","Type":"ContainerStarted","Data":"fedc24105f934e76b325a21713e154a00e668f283ff834371fd1929891a60c8a"} Nov 02 19:58:11.980680 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:11.980643 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:11.981085 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:11.980743 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:11.981085 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:11.980792 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:12.919868 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:12.919822 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" event={"ID":"7f8ffee0-608f-4c70-bbfc-6d1f34a53460","Type":"ContainerStarted","Data":"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967"} Nov 02 19:58:13.126155 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.126097 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podStartSLOduration=3.173383719 podStartE2EDuration="7.126080723s" totalImagesPullingTime="3.952697004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:13.077565973 +0000 UTC m=+576.684110470" watchObservedRunningTime="2026-11-02 19:58:13.126080723 +0000 UTC m=+576.732625217" Nov 02 19:58:13.521184 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.521147 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:13.522824 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.522798 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:13.522961 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.522873 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:13.626122 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.626085 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:13.638876 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.638848 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:13.935379 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:13.935305 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4lsbj" Nov 02 19:58:14.521947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:14.521908 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:14.522425 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:14.521971 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:15.521005 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:15.520972 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:15.521174 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:15.521020 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:16.521304 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:16.521260 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:16.521714 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:16.521319 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:17.521450 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:17.521413 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:17.521949 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:17.521482 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:18.521979 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:18.521942 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:58:18.522363 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:18.521994 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 19:58:20.522221 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:20.522170 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:58:20.522628 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:20.522240 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:21.007980 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:58:21.007945 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:58:21.563879 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:21.563846 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:58:21.564270 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:21.563896 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:21.645093 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:21.645061 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:21.645810 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:21.645776 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:21.646523 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:21.646499 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 19:58:22.025187 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.025160 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 19:58:22.028473 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.028452 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.110966 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.110937 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 19:58:22.118558 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118533 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118692 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118565 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118692 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118648 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118717 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6ck\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118786 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118825 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118809 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118974 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118904 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.118974 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118965 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.119054 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.118991 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219460 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219428 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219460 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219461 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6ck\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219685 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219481 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219769 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219749 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219842 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219824 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219907 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219852 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219907 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219869 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219907 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219887 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.219907 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.219905 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.220114 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.220054 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.220164 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.220116 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.220246 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.220229 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.220363 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.220341 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.220421 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.220399 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.222245 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.222220 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.222477 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.222460 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.274457 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.274427 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.274595 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.274494 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6ck\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2hpkp\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.344877 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.344804 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:22.756932 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.756912 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 19:58:22.758784 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:58:22.758755 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53758626_0b3c_4e81_9809_b8df32c09757.slice/crio-e77882c470d849e0a3814f4aee467163ca420826ecf248ed344e4eea60c1ef57 WatchSource:0}: Error finding container e77882c470d849e0a3814f4aee467163ca420826ecf248ed344e4eea60c1ef57: Status 404 returned error can't find the container with id e77882c470d849e0a3814f4aee467163ca420826ecf248ed344e4eea60c1ef57 Nov 02 19:58:22.762168 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.762132 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:22.762265 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.762216 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:22.762327 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.762263 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:22.969870 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.969832 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" event={"ID":"53758626-0b3c-4e81-9809-b8df32c09757","Type":"ContainerStarted","Data":"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b"} Nov 02 19:58:22.970056 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:22.969898 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" event={"ID":"53758626-0b3c-4e81-9809-b8df32c09757","Type":"ContainerStarted","Data":"e77882c470d849e0a3814f4aee467163ca420826ecf248ed344e4eea60c1ef57"} Nov 02 19:58:23.106264 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:23.106166 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podStartSLOduration=2.106150275 podStartE2EDuration="2.106150275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:23.104000114 +0000 UTC m=+586.710544612" watchObservedRunningTime="2026-11-02 19:58:23.106150275 +0000 UTC m=+586.712694772" Nov 02 19:58:23.345359 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:23.345327 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:23.346846 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:23.346820 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:23.346903 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:23.346873 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:24.345975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:24.345943 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:24.346399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:24.345994 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:25.345241 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:25.345205 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:25.345413 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:25.345261 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:26.345408 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:26.345372 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:26.345856 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:26.345423 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:27.345186 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:27.345145 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:27.345378 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:27.345204 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:28.345383 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:28.345342 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:28.345883 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:28.345403 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:29.346004 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:29.345967 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:58:29.346398 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:29.346024 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 19:58:31.013280 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:31.013243 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:31.013767 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:31.013606 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:31.014620 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:58:31.014594 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 19:58:51.003207 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:58:51.003171 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:59:13.737008 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.736974 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 19:59:13.741039 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.741017 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.797841 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.797944 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.797989 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798016 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798044 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhmd\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798085 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798131 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.798403 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.798198 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.807234 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.807199 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3acf2229-openshift-default-dockercfg-7sl7j\"" Nov 02 19:59:13.845861 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.845830 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 19:59:13.898908 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.898876 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.898908 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.898908 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.898926 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhmd\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.898954 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.898991 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899012 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899032 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899160 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899081 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899448 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899178 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899500 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899453 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899556 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899530 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899678 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899657 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899678 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899671 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.899886 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.899864 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.901334 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.901308 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.901785 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.901767 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.954025 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.953994 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:13.954363 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:13.954338 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhmd\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\") pod \"e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:14.053709 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.053567 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:14.475147 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.475123 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 19:59:14.477566 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:14.477535 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f74ef6e_6b06_4701_b0ce_2cbc6c4552f1.slice/crio-0a3d780b5fd0ca25c8af0ff33d7ec29e9f8d8066cedb5f1410e0819682d68262 WatchSource:0}: Error finding container 0a3d780b5fd0ca25c8af0ff33d7ec29e9f8d8066cedb5f1410e0819682d68262: Status 404 returned error can't find the container with id 0a3d780b5fd0ca25c8af0ff33d7ec29e9f8d8066cedb5f1410e0819682d68262 Nov 02 19:59:14.479988 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.479953 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:14.480071 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.480019 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:14.480071 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.480047 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:14.676593 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.676560 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 19:59:14.680292 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.680262 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.733688 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.733617 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a0d19e8c-gw-openshift-default-dockercfg-7hxx5\"" Nov 02 19:59:14.736622 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.735654 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 19:59:14.814375 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814339 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814483 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814545 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814570 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjp9\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814625 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814656 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814683 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814798 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.814916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.814826 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.915858 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.915827 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916008 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.915883 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916008 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.915908 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjp9\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916008 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.915938 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916105 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916057 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916140 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916111 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916197 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916160 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916262 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916196 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916262 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916237 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916430 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916405 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916524 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916501 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916582 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916568 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916648 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916629 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.916688 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.916628 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.918061 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.918040 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.918364 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.918348 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.969812 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.969790 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjp9\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.969937 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.969789 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\") pod \"e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:14.995544 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:14.995474 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:15.227392 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.227354 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerStarted","Data":"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2"} Nov 02 19:59:15.227551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.227416 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerStarted","Data":"0a3d780b5fd0ca25c8af0ff33d7ec29e9f8d8066cedb5f1410e0819682d68262"} Nov 02 19:59:15.346294 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.346178 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podStartSLOduration=2.346160578 podStartE2EDuration="2.346160578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:15.345687948 +0000 UTC m=+638.952232447" watchObservedRunningTime="2026-11-02 19:59:15.346160578 +0000 UTC m=+638.952705077" Nov 02 19:59:15.469626 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.469575 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 19:59:15.484744 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:15.484696 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abaf25f_2e4e_4d8d_92db_c6fc52297d51.slice/crio-2f9b7027d26dfee6e4ed05fb7bb2aaac60a9696ab8dfe6ed2358545c2aae1a49 WatchSource:0}: Error finding container 2f9b7027d26dfee6e4ed05fb7bb2aaac60a9696ab8dfe6ed2358545c2aae1a49: Status 404 returned error can't find the container with id 2f9b7027d26dfee6e4ed05fb7bb2aaac60a9696ab8dfe6ed2358545c2aae1a49 Nov 02 19:59:15.490570 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.490534 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:15.490655 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.490605 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:15.490655 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:15.490633 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:16.054658 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.054617 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:16.059461 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.059439 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:16.234342 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.234305 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerStarted","Data":"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198"} Nov 02 19:59:16.234503 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.234366 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerStarted","Data":"2f9b7027d26dfee6e4ed05fb7bb2aaac60a9696ab8dfe6ed2358545c2aae1a49"} Nov 02 19:59:16.234596 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.234521 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:16.235493 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.235474 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 19:59:16.364793 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.364670 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podStartSLOduration=2.364656544 podStartE2EDuration="2.364656544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:16.363982501 +0000 UTC m=+639.970526999" watchObservedRunningTime="2026-11-02 19:59:16.364656544 +0000 UTC m=+639.971201042" Nov 02 19:59:16.996023 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:16.995990 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:17.001039 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:17.001016 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:17.239666 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:17.239633 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:17.240603 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:17.240573 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:20.456487 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.456453 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 19:59:20.459991 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.459971 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.510059 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.510029 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-172adb07-openshift-default-dockercfg-qzs64\"" Nov 02 19:59:20.560247 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.560216 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 19:59:20.572947 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.572919 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.572961 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.572979 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573082 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573219 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573112 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573219 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573161 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573279 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573225 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573312 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573280 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhhq\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.573312 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.573304 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.674692 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674660 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.674914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674712 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.674914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674779 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.674914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674863 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.674914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674888 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhhq\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675125 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674923 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675125 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.674985 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675164 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675181 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675217 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675207 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675364 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675216 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675437 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675413 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675496 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675476 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.675768 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.675748 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.677220 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.677199 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.677297 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.677279 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.729791 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.729684 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.730683 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.730654 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhhq\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\") pod \"e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:20.773485 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:20.773452 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:21.006419 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:21.006347 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:59:21.189290 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.189222 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 19:59:21.191783 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:21.191752 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c51300_cb4c_4584_a29e_9fcdcea9e916.slice/crio-a5243a183b02ea2a7e5278c3e65a6c1c1ac2dd54c69635f80470ce95a5974f44 WatchSource:0}: Error finding container a5243a183b02ea2a7e5278c3e65a6c1c1ac2dd54c69635f80470ce95a5974f44: Status 404 returned error can't find the container with id a5243a183b02ea2a7e5278c3e65a6c1c1ac2dd54c69635f80470ce95a5974f44 Nov 02 19:59:21.194047 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.194015 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:21.194111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.194079 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:21.194111 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.194108 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:21.260933 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.260857 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerStarted","Data":"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0"} Nov 02 19:59:21.261060 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.260931 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerStarted","Data":"a5243a183b02ea2a7e5278c3e65a6c1c1ac2dd54c69635f80470ce95a5974f44"} Nov 02 19:59:21.372234 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.372178 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podStartSLOduration=1.3721654810000001 podStartE2EDuration="1.372165481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:21.370972044 +0000 UTC m=+644.977516544" watchObservedRunningTime="2026-11-02 19:59:21.372165481 +0000 UTC m=+644.978709976" Nov 02 19:59:21.774001 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:21.773968 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:22.779572 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:22.779543 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:23.269376 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:23.269346 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:23.270253 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:23.270234 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 19:59:23.954814 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:23.954783 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 19:59:23.958560 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:23.958539 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.010989 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.010967 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4db200eb-serving-cert\"" Nov 02 19:59:24.013056 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.013034 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 19:59:24.117894 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.117865 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.118048 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.117911 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7j2\" (UniqueName: \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.118048 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.118032 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.118177 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.118160 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.219381 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.219292 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.219706 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.219680 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.220859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.219905 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.220859 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.219964 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7j2\" (UniqueName: \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.229450 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.229421 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.230711 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.230685 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.231216 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.231195 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.274958 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.274927 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7j2\" (UniqueName: \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\") pod \"maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.571580 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.571505 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:24.934329 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.934308 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 19:59:24.935511 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:24.935481 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e71c172_dc08_465c_9689_55d748dc8a60.slice/crio-24d845c6dd078cea3e72eaccd397a2cc76f7a52042f0b4a4c344619d41e328ac WatchSource:0}: Error finding container 24d845c6dd078cea3e72eaccd397a2cc76f7a52042f0b4a4c344619d41e328ac: Status 404 returned error can't find the container with id 24d845c6dd078cea3e72eaccd397a2cc76f7a52042f0b4a4c344619d41e328ac Nov 02 19:59:24.937165 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:24.937147 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:59:25.281216 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:25.281176 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" event={"ID":"2e71c172-dc08-465c-9689-55d748dc8a60","Type":"ContainerStarted","Data":"24d845c6dd078cea3e72eaccd397a2cc76f7a52042f0b4a4c344619d41e328ac"} Nov 02 19:59:26.287503 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:26.287467 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" event={"ID":"2e71c172-dc08-465c-9689-55d748dc8a60","Type":"ContainerStarted","Data":"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c"} Nov 02 19:59:26.287914 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:26.287549 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:28.855675 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.855606 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" podStartSLOduration=4.679979695 podStartE2EDuration="5.855590376s" totalImagesPullingTime="1.175610681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:26.40079166 +0000 UTC m=+650.007336163" watchObservedRunningTime="2026-11-02 19:59:28.855590376 +0000 UTC m=+652.462134874" Nov 02 19:59:28.856187 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.855998 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 19:59:28.859828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.859807 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:28.961539 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.961487 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4db200eb\"" Nov 02 19:59:28.963006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.962965 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4db200eb-dockercfg-d8rfb\"" Nov 02 19:59:28.965659 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.965631 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 19:59:28.978656 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.978627 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2z2\" (UniqueName: \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:28.978790 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:28.978676 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.080020 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.079989 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2z2\" (UniqueName: \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.080020 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.080022 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.080755 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.080709 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.135153 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.135061 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2z2\" (UniqueName: \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\") pod \"payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.172708 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.172677 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:29.278829 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.278794 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 19:59:29.286911 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.286884 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.335902 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.335873 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 19:59:29.373420 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.373389 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-70ebea5b-serving-cert\"" Nov 02 19:59:29.383921 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.383897 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.384026 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.383949 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.384093 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.384054 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxkf\" (UniqueName: \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.384207 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.384190 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.485262 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.485232 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.485430 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.485283 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.485430 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.485338 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.485430 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:29.485385 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-serving-cert: secret "maas-api-e2e-ait-70ebea5b-serving-cert" not found Nov 02 19:59:29.485430 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.485402 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxkf\" (UniqueName: \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.485602 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:29.485450 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls podName:04a592e0-1d82-45da-a7f1-5b2a3fe3d350 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:29.98542878 +0000 UTC m=+653.591973263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls") pod "maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" (UID: "04a592e0-1d82-45da-a7f1-5b2a3fe3d350") : secret "maas-api-e2e-ait-70ebea5b-serving-cert" not found Nov 02 19:59:29.486064 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.486043 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.487903 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.487882 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.545527 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.545183 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 19:59:29.546701 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.546672 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxkf\" (UniqueName: \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.550441 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:29.549853 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080710a6_b6c7_4838_9d64_df3ce37b4c00.slice/crio-bf48e2c8e276b5c976818ccd215f36b6345132518018b791bb7fe648a4e13887 WatchSource:0}: Error finding container bf48e2c8e276b5c976818ccd215f36b6345132518018b791bb7fe648a4e13887: Status 404 returned error can't find the container with id bf48e2c8e276b5c976818ccd215f36b6345132518018b791bb7fe648a4e13887 Nov 02 19:59:29.991536 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.991506 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:29.993837 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:29.993818 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") pod \"maas-api-e2e-ait-70ebea5b-85b94b96-79xtn\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:30.201024 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.200989 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:30.309234 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.309201 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" event={"ID":"080710a6-b6c7-4838-9d64-df3ce37b4c00","Type":"ContainerStarted","Data":"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf"} Nov 02 19:59:30.309415 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.309254 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" event={"ID":"080710a6-b6c7-4838-9d64-df3ce37b4c00","Type":"ContainerStarted","Data":"bf48e2c8e276b5c976818ccd215f36b6345132518018b791bb7fe648a4e13887"} Nov 02 19:59:30.309415 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.309322 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:30.419047 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.418990 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" podStartSLOduration=2.418976816 podStartE2EDuration="2.418976816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:30.416995486 +0000 UTC m=+654.023539984" watchObservedRunningTime="2026-11-02 19:59:30.418976816 +0000 UTC m=+654.025521308" Nov 02 19:59:30.617362 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:30.617330 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 19:59:30.617960 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:30.617935 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a592e0_1d82_45da_a7f1_5b2a3fe3d350.slice/crio-9b2933e13763da09b8d3e4365c5fed9797d49b9a66d1559746265fab0ad1f627 WatchSource:0}: Error finding container 9b2933e13763da09b8d3e4365c5fed9797d49b9a66d1559746265fab0ad1f627: Status 404 returned error can't find the container with id 9b2933e13763da09b8d3e4365c5fed9797d49b9a66d1559746265fab0ad1f627 Nov 02 19:59:31.316530 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:31.316498 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" event={"ID":"04a592e0-1d82-45da-a7f1-5b2a3fe3d350","Type":"ContainerStarted","Data":"9b2933e13763da09b8d3e4365c5fed9797d49b9a66d1559746265fab0ad1f627"} Nov 02 19:59:32.297950 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:32.297922 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 19:59:32.322950 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:32.322920 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" event={"ID":"04a592e0-1d82-45da-a7f1-5b2a3fe3d350","Type":"ContainerStarted","Data":"b927808ed0b2b2447f20feb2a022e5f8c7dadfbe7aad62330c05d327c65bcf29"} Nov 02 19:59:32.323298 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:32.323007 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:32.516157 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:32.516087 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" podStartSLOduration=2.64027114 podStartE2EDuration="3.516067955s" totalImagesPullingTime="875.796815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:32.514533727 +0000 UTC m=+656.121078231" watchObservedRunningTime="2026-11-02 19:59:32.516067955 +0000 UTC m=+656.122612455" Nov 02 19:59:34.649385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.649350 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 19:59:34.653318 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.653297 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.703916 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.703889 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a0d19e8c-serving-cert\"" Nov 02 19:59:34.746488 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.746463 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.746622 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.746497 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.746622 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.746611 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.746689 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.746672 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfw4\" (UniqueName: \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.752938 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.752917 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 19:59:34.847624 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.847591 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.847804 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.847690 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.847804 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.847757 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfw4\" (UniqueName: \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.847933 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.847855 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.847988 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:34.847965 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-serving-cert: secret "maas-api-e2e-migrate-a0d19e8c-serving-cert" not found Nov 02 19:59:34.848057 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:34.848042 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls podName:3cc809ad-632e-41a5-a022-486056694c47 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:35.348019477 +0000 UTC m=+658.954563954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls") pod "maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" (UID: "3cc809ad-632e-41a5-a022-486056694c47") : secret "maas-api-e2e-migrate-a0d19e8c-serving-cert" not found Nov 02 19:59:34.848399 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.848380 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.850116 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.850093 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:34.922115 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:34.922041 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfw4\" (UniqueName: \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:35.351621 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:35.351586 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:35.354001 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:35.353977 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:35.566278 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:35.566246 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:35.930002 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:35.929981 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 19:59:35.933151 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:35.933121 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc809ad_632e_41a5_a022_486056694c47.slice/crio-e33e233edeccd1442ca33165195813efd179bc60ff71ef111945450d528055b2 WatchSource:0}: Error finding container e33e233edeccd1442ca33165195813efd179bc60ff71ef111945450d528055b2: Status 404 returned error can't find the container with id e33e233edeccd1442ca33165195813efd179bc60ff71ef111945450d528055b2 Nov 02 19:59:36.346681 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:36.346652 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" event={"ID":"3cc809ad-632e-41a5-a022-486056694c47","Type":"ContainerStarted","Data":"e33e233edeccd1442ca33165195813efd179bc60ff71ef111945450d528055b2"} Nov 02 19:59:37.360136 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:37.360098 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" event={"ID":"3cc809ad-632e-41a5-a022-486056694c47","Type":"ContainerStarted","Data":"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb"} Nov 02 19:59:37.360521 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:37.360235 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:37.468468 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:37.468422 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" podStartSLOduration=2.702021291 podStartE2EDuration="3.468409879s" totalImagesPullingTime="766.388588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:37.468085448 +0000 UTC m=+661.074629947" watchObservedRunningTime="2026-11-02 19:59:37.468409879 +0000 UTC m=+661.074954377" Nov 02 19:59:38.335202 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:38.335171 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 19:59:41.320006 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:41.319981 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 19:59:43.371602 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:43.371572 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 19:59:44.987414 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:44.987381 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 19:59:45.007989 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.007960 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.041635 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.041610 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.041778 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.041664 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.041778 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.041684 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.041778 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.041710 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2c6\" (UniqueName: \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.043363 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.043327 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 19:59:45.100966 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.100937 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3acf2229-serving-cert\"" Nov 02 19:59:45.143894 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.143864 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.144149 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.144122 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.144285 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.144268 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.144428 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.144411 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2c6\" (UniqueName: \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.144772 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:45.144021 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-serving-cert: secret "maas-api-e2e-ait-3acf2229-serving-cert" not found Nov 02 19:59:45.144864 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:45.144835 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls podName:dca9f0d9-6f8c-42dc-97a5-328ee32c9b29 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:45.644813086 +0000 UTC m=+669.251357567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls") pod "maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" (UID: "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29") : secret "maas-api-e2e-ait-3acf2229-serving-cert" not found Nov 02 19:59:45.144936 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.144891 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.148084 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.148061 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.217798 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.217765 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2c6\" (UniqueName: \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.652609 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.652566 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.655188 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.655163 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") pod \"maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:45.924799 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:45.924702 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:46.305088 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:46.304970 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 19:59:46.316993 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:46.316847 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca9f0d9_6f8c_42dc_97a5_328ee32c9b29.slice/crio-aab1974fd0ebaa165813a1dba8cd6c705e2fcbe7dc08577574b1b11d0e94847b WatchSource:0}: Error finding container aab1974fd0ebaa165813a1dba8cd6c705e2fcbe7dc08577574b1b11d0e94847b: Status 404 returned error can't find the container with id aab1974fd0ebaa165813a1dba8cd6c705e2fcbe7dc08577574b1b11d0e94847b Nov 02 19:59:46.427327 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:46.427292 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" event={"ID":"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29","Type":"ContainerStarted","Data":"aab1974fd0ebaa165813a1dba8cd6c705e2fcbe7dc08577574b1b11d0e94847b"} Nov 02 19:59:47.433694 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:47.433657 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" event={"ID":"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29","Type":"ContainerStarted","Data":"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd"} Nov 02 19:59:47.434087 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:47.433887 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:47.558219 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:47.558110 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" podStartSLOduration=2.658677846 podStartE2EDuration="3.558096316s" totalImagesPullingTime="899.41847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:47.557201853 +0000 UTC m=+671.163746351" watchObservedRunningTime="2026-11-02 19:59:47.558096316 +0000 UTC m=+671.164640810" Nov 02 19:59:49.963963 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.963929 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 19:59:49.968410 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.968383 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:49.998813 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.998788 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:49.998933 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.998821 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:49.998933 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.998848 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxjq\" (UniqueName: \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:49.999053 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:49.999003 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.018398 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.018375 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-172adb07-serving-cert\"" Nov 02 19:59:50.068761 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.068707 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 19:59:50.100135 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.100104 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.100315 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.100226 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.100315 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.100248 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.100315 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.100271 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxjq\" (UniqueName: \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.100543 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:50.100477 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-serving-cert: secret "maas-api-e2e-derive-172adb07-serving-cert" not found Nov 02 19:59:50.100601 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:50.100557 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls podName:dfc1a814-aa10-4c18-9097-de7c82467610 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:50.600538412 +0000 UTC m=+674.207082889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls") pod "maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" (UID: "dfc1a814-aa10-4c18-9097-de7c82467610") : secret "maas-api-e2e-derive-172adb07-serving-cert" not found Nov 02 19:59:50.101090 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.101067 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.103102 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.103072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.155173 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.155143 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxjq\" (UniqueName: \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.606433 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.606398 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.608955 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.608928 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") pod \"maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:50.882128 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:50.882058 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:51.004204 ip-10-0-128-34 kubenswrapper[5136]: E1102 19:59:51.004170 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 19:59:51.244975 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.244947 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 19:59:51.246708 ip-10-0-128-34 kubenswrapper[5136]: W1102 19:59:51.246680 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1a814_aa10_4c18_9097_de7c82467610.slice/crio-87b61a35102cf28ef4379381f4df8c2ec45dd43d4e75406b31d6716f52bc5569 WatchSource:0}: Error finding container 87b61a35102cf28ef4379381f4df8c2ec45dd43d4e75406b31d6716f52bc5569: Status 404 returned error can't find the container with id 87b61a35102cf28ef4379381f4df8c2ec45dd43d4e75406b31d6716f52bc5569 Nov 02 19:59:51.456516 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.456484 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" event={"ID":"dfc1a814-aa10-4c18-9097-de7c82467610","Type":"ContainerStarted","Data":"87b61a35102cf28ef4379381f4df8c2ec45dd43d4e75406b31d6716f52bc5569"} Nov 02 19:59:51.458393 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.458365 5136 generic.go:356] "Generic (PLEG): container finished" podID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerID="3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198" exitCode=0 Nov 02 19:59:51.458515 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.458411 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerDied","Data":"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198"} Nov 02 19:59:51.458828 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.458812 5136 scope.go:122] "RemoveContainer" containerID="3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198" Nov 02 19:59:51.460373 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.460346 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:51.460439 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.460414 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:51.460500 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:51.460445 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:52.465630 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.465437 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" event={"ID":"dfc1a814-aa10-4c18-9097-de7c82467610","Type":"ContainerStarted","Data":"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39"} Nov 02 19:59:52.466288 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.466262 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 19:59:52.468145 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.468119 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerStarted","Data":"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad"} Nov 02 19:59:52.612755 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.612185 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" podStartSLOduration=2.714443525 podStartE2EDuration="3.612163981s" totalImagesPullingTime="897.720456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:52.612010894 +0000 UTC m=+676.218555393" watchObservedRunningTime="2026-11-02 19:59:52.612163981 +0000 UTC m=+676.218708480" Nov 02 19:59:52.996082 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.996050 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:52.996441 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.996417 5136 patch_prober.go:29] interesting pod/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 19:59:52.996621 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:52.996596 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 19:59:53.445385 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:53.445308 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 19:59:53.997101 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:53.997064 5136 patch_prober.go:29] interesting pod/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 19:59:53.997811 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:53.997780 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 19:59:55.005755 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:54.996048 5136 patch_prober.go:29] interesting pod/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 19:59:55.005755 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:54.996105 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 19:59:55.996551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:55.996099 5136 patch_prober.go:29] interesting pod/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 19:59:55.996551 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:55.996171 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 19:59:57.002162 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:57.002118 5136 patch_prober.go:29] interesting pod/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 19:59:57.002788 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:57.002752 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 19:59:58.000943 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:58.000911 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:58.001426 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:58.001387 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:58.002301 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:58.002276 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 19:59:59.483440 ip-10-0-128-34 kubenswrapper[5136]: I1102 19:59:59.483414 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 20:00:01.440161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.440091 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:46694->10.133.0.42:15021: read: connection reset by peer" start-of-body= Nov 02 20:00:01.440603 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.440174 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:46694->10.133.0.42:15021: read: connection reset by peer" Nov 02 20:00:01.521327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.521294 5136 generic.go:356] "Generic (PLEG): container finished" podID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerID="f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2" exitCode=0 Nov 02 20:00:01.521489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.521372 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerDied","Data":"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2"} Nov 02 20:00:01.521822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.521803 5136 scope.go:122] "RemoveContainer" containerID="f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2" Nov 02 20:00:01.523136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.523105 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:01.523210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.523174 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:01.523210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:01.523204 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:02.547124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:02.547089 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerStarted","Data":"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95"} Nov 02 20:00:03.054167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:03.054129 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:03.054349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:03.054327 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 20:00:03.054403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:03.054386 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 20:00:04.054001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:04.053967 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 20:00:04.054406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:04.054020 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 20:00:05.054393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:05.054353 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 20:00:05.054850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:05.054404 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 20:00:06.055040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:06.055010 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 20:00:06.055475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:06.055063 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 20:00:07.054450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:07.054410 5136 patch_prober.go:29] interesting pod/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 20:00:07.054648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:07.054468 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 20:00:08.058875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:08.058841 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:08.059359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:08.059340 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:08.059991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:08.059975 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:09.310921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:09.310878 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:00:09.311393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:09.310964 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:00:10.585543 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.585506 5136 generic.go:356] "Generic (PLEG): container finished" podID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerID="f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0" exitCode=0 Nov 02 20:00:10.585988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.585549 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerDied","Data":"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0"} Nov 02 20:00:10.585988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.585882 5136 scope.go:122] "RemoveContainer" containerID="f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0" Nov 02 20:00:10.587047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.587017 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:10.587122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.587085 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:10.587158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:10.587118 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:11.615261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:11.615225 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerStarted","Data":"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb"} Nov 02 20:00:11.774359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:11.774323 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:00:11.774578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:11.774555 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 20:00:11.774660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:11.774608 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 20:00:12.774873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:12.774835 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 20:00:12.775267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:12.774903 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 20:00:13.774336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:13.774300 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 20:00:13.774548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:13.774359 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 20:00:14.773846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:14.773807 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 20:00:14.774299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:14.773862 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 20:00:15.774747 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:15.774710 5136 patch_prober.go:29] interesting pod/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 20:00:15.775185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:15.774774 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 20:00:16.033186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:16.033097 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 20:00:16.778511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:16.778478 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:00:16.778906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:16.778848 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:00:16.779429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:16.779409 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:00:21.007879 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:21.007838 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:00:21.034473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.034427 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" podUID="080710a6-b6c7-4838-9d64-df3ce37b4c00" containerName="payload-pre-processing" containerID="cri-o://006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf" gracePeriod=25 Nov 02 20:00:21.284534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.284512 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 20:00:21.429838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.429811 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\" (UniqueName: \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\") pod \"080710a6-b6c7-4838-9d64-df3ce37b4c00\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " Nov 02 20:00:21.429999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.429936 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\") pod \"080710a6-b6c7-4838-9d64-df3ce37b4c00\" (UID: \"080710a6-b6c7-4838-9d64-df3ce37b4c00\") " Nov 02 20:00:21.430318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.430285 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume" pod "080710a6-b6c7-4838-9d64-df3ce37b4c00" (UID: "080710a6-b6c7-4838-9d64-df3ce37b4c00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:21.431794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.431768 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2" pod "080710a6-b6c7-4838-9d64-df3ce37b4c00" (UID: "080710a6-b6c7-4838-9d64-df3ce37b4c00"). InnerVolumeSpecName "kube-api-access-5w2z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:21.530694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.530614 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080710a6-b6c7-4838-9d64-df3ce37b4c00-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.530694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.530644 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w2z2\" (UniqueName: \"kubernetes.io/projected/080710a6-b6c7-4838-9d64-df3ce37b4c00-kube-api-access-5w2z2\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.668333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.668299 5136 generic.go:356] "Generic (PLEG): container finished" podID="080710a6-b6c7-4838-9d64-df3ce37b4c00" containerID="006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf" exitCode=0 Nov 02 20:00:21.668505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.668360 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" Nov 02 20:00:21.668505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.668353 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" event={"ID":"080710a6-b6c7-4838-9d64-df3ce37b4c00","Type":"ContainerDied","Data":"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf"} Nov 02 20:00:21.668505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.668478 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh" event={"ID":"080710a6-b6c7-4838-9d64-df3ce37b4c00","Type":"ContainerDied","Data":"bf48e2c8e276b5c976818ccd215f36b6345132518018b791bb7fe648a4e13887"} Nov 02 20:00:21.668628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.668507 5136 scope.go:122] "RemoveContainer" containerID="006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf" Nov 02 20:00:21.686113 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.686094 5136 scope.go:122] "RemoveContainer" containerID="006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf" Nov 02 20:00:21.686418 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:21.686399 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf\": container with ID starting with 006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf not found: ID does not exist" containerID="006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf" Nov 02 20:00:21.686460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.686428 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf"} err="failed to get container status \"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf\": rpc error: code = NotFound desc = could not find container \"006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf\": container with ID starting with 006070a0de27e0e31bf978f335970c52d1a10f99e2b4939ab2744dfa26eb27bf not found: ID does not exist" Nov 02 20:00:21.824374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.824295 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 20:00:21.827374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:21.827351 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4db200eb-7b45467b7f-f2lrh"] Nov 02 20:00:22.545714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:22.545672 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080710a6-b6c7-4838-9d64-df3ce37b4c00" path="/var/lib/kubelet/pods/080710a6-b6c7-4838-9d64-df3ce37b4c00/volumes" Nov 02 20:00:26.266149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.266114 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 20:00:26.266710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.266317 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" podUID="2e71c172-dc08-465c-9689-55d748dc8a60" containerName="maas-api" containerID="cri-o://8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c" gracePeriod=30 Nov 02 20:00:26.557952 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.557921 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 20:00:26.601949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.601914 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\") pod \"2e71c172-dc08-465c-9689-55d748dc8a60\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " Nov 02 20:00:26.602153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.602028 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\") pod \"2e71c172-dc08-465c-9689-55d748dc8a60\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " Nov 02 20:00:26.602153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.602094 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\") pod \"2e71c172-dc08-465c-9689-55d748dc8a60\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " Nov 02 20:00:26.602279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.602155 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\" (UniqueName: \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\") pod \"2e71c172-dc08-465c-9689-55d748dc8a60\" (UID: \"2e71c172-dc08-465c-9689-55d748dc8a60\") " Nov 02 20:00:26.602345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.602295 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca" pod "2e71c172-dc08-465c-9689-55d748dc8a60" (UID: "2e71c172-dc08-465c-9689-55d748dc8a60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:26.602631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.602582 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e71c172-dc08-465c-9689-55d748dc8a60-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:26.604288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.604264 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2" pod "2e71c172-dc08-465c-9689-55d748dc8a60" (UID: "2e71c172-dc08-465c-9689-55d748dc8a60"). InnerVolumeSpecName "kube-api-access-jq7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:26.604411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.604288 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs" pod "2e71c172-dc08-465c-9689-55d748dc8a60" (UID: "2e71c172-dc08-465c-9689-55d748dc8a60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:26.604411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.604298 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls" pod "2e71c172-dc08-465c-9689-55d748dc8a60" (UID: "2e71c172-dc08-465c-9689-55d748dc8a60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:26.698877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.698843 5136 generic.go:356] "Generic (PLEG): container finished" podID="2e71c172-dc08-465c-9689-55d748dc8a60" containerID="8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c" exitCode=0 Nov 02 20:00:26.699072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.698898 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" Nov 02 20:00:26.699072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.698898 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" event={"ID":"2e71c172-dc08-465c-9689-55d748dc8a60","Type":"ContainerDied","Data":"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c"} Nov 02 20:00:26.699072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.699013 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r" event={"ID":"2e71c172-dc08-465c-9689-55d748dc8a60","Type":"ContainerDied","Data":"24d845c6dd078cea3e72eaccd397a2cc76f7a52042f0b4a4c344619d41e328ac"} Nov 02 20:00:26.699072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.699036 5136 scope.go:122] "RemoveContainer" containerID="8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c" Nov 02 20:00:26.703291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.703263 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:26.703413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.703293 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e71c172-dc08-465c-9689-55d748dc8a60-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:26.703413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.703310 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7j2\" (UniqueName: \"kubernetes.io/projected/2e71c172-dc08-465c-9689-55d748dc8a60-kube-api-access-jq7j2\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:26.714694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.714666 5136 scope.go:122] "RemoveContainer" containerID="8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c" Nov 02 20:00:26.715002 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:26.714982 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c\": container with ID starting with 8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c not found: ID does not exist" containerID="8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c" Nov 02 20:00:26.715079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.715014 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c"} err="failed to get container status \"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c\": rpc error: code = NotFound desc = could not find container \"8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c\": container with ID starting with 8d75ea5076fec8f2756c3f457157e7c2137196e8d9c41662814c6533563e943c not found: ID does not exist" Nov 02 20:00:26.876628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.876539 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 20:00:26.879332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:26.879283 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4db200eb-56d49fbf6b-k9b8r"] Nov 02 20:00:27.559105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.559071 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 20:00:27.559551 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.559248 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" podUID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" containerName="maas-api" containerID="cri-o://b927808ed0b2b2447f20feb2a022e5f8c7dadfbe7aad62330c05d327c65bcf29" gracePeriod=30 Nov 02 20:00:27.707519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.707481 5136 generic.go:356] "Generic (PLEG): container finished" podID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" containerID="b927808ed0b2b2447f20feb2a022e5f8c7dadfbe7aad62330c05d327c65bcf29" exitCode=0 Nov 02 20:00:27.707750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.707560 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" event={"ID":"04a592e0-1d82-45da-a7f1-5b2a3fe3d350","Type":"ContainerDied","Data":"b927808ed0b2b2447f20feb2a022e5f8c7dadfbe7aad62330c05d327c65bcf29"} Nov 02 20:00:27.831829 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.831804 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 20:00:27.915596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.915562 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\") pod \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " Nov 02 20:00:27.915596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.915601 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\") pod \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " Nov 02 20:00:27.915888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.915649 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\" (UniqueName: \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\") pod \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " Nov 02 20:00:27.915888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.915719 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") pod \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\" (UID: \"04a592e0-1d82-45da-a7f1-5b2a3fe3d350\") " Nov 02 20:00:27.916052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.916026 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca" pod "04a592e0-1d82-45da-a7f1-5b2a3fe3d350" (UID: "04a592e0-1d82-45da-a7f1-5b2a3fe3d350"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:27.917924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.917895 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf" pod "04a592e0-1d82-45da-a7f1-5b2a3fe3d350" (UID: "04a592e0-1d82-45da-a7f1-5b2a3fe3d350"). InnerVolumeSpecName "kube-api-access-bnxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:27.918022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.917930 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs" pod "04a592e0-1d82-45da-a7f1-5b2a3fe3d350" (UID: "04a592e0-1d82-45da-a7f1-5b2a3fe3d350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:27.918022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:27.917962 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls" pod "04a592e0-1d82-45da-a7f1-5b2a3fe3d350" (UID: "04a592e0-1d82-45da-a7f1-5b2a3fe3d350"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:28.016922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.016888 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.016922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.016922 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxkf\" (UniqueName: \"kubernetes.io/projected/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-kube-api-access-bnxkf\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.017157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.016937 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.017157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.016949 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a592e0-1d82-45da-a7f1-5b2a3fe3d350-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.545444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.545364 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e71c172-dc08-465c-9689-55d748dc8a60" path="/var/lib/kubelet/pods/2e71c172-dc08-465c-9689-55d748dc8a60/volumes" Nov 02 20:00:28.715947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.715920 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" Nov 02 20:00:28.715947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.715926 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn" event={"ID":"04a592e0-1d82-45da-a7f1-5b2a3fe3d350","Type":"ContainerDied","Data":"9b2933e13763da09b8d3e4365c5fed9797d49b9a66d1559746265fab0ad1f627"} Nov 02 20:00:28.716458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.715999 5136 scope.go:122] "RemoveContainer" containerID="b927808ed0b2b2447f20feb2a022e5f8c7dadfbe7aad62330c05d327c65bcf29" Nov 02 20:00:28.872563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.872474 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 20:00:28.920475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.920441 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-70ebea5b-85b94b96-79xtn"] Nov 02 20:00:28.923906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.923876 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 20:00:28.924324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:28.924292 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" podUID="3cc809ad-632e-41a5-a022-486056694c47" containerName="maas-api" containerID="cri-o://2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb" gracePeriod=30 Nov 02 20:00:29.192283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.192260 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 20:00:29.242352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.242323 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\" (UniqueName: \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\") pod \"3cc809ad-632e-41a5-a022-486056694c47\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " Nov 02 20:00:29.242498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.242368 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\") pod \"3cc809ad-632e-41a5-a022-486056694c47\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " Nov 02 20:00:29.242498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.242400 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\") pod \"3cc809ad-632e-41a5-a022-486056694c47\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " Nov 02 20:00:29.242498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.242461 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") pod \"3cc809ad-632e-41a5-a022-486056694c47\" (UID: \"3cc809ad-632e-41a5-a022-486056694c47\") " Nov 02 20:00:29.242806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.242786 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca" pod "3cc809ad-632e-41a5-a022-486056694c47" (UID: "3cc809ad-632e-41a5-a022-486056694c47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:29.244469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.244440 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs" pod "3cc809ad-632e-41a5-a022-486056694c47" (UID: "3cc809ad-632e-41a5-a022-486056694c47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:29.244577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.244510 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls" pod "3cc809ad-632e-41a5-a022-486056694c47" (UID: "3cc809ad-632e-41a5-a022-486056694c47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:29.244712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.244690 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4" pod "3cc809ad-632e-41a5-a022-486056694c47" (UID: "3cc809ad-632e-41a5-a022-486056694c47"). InnerVolumeSpecName "kube-api-access-svfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:29.343656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.343613 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:29.343656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.343640 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc809ad-632e-41a5-a022-486056694c47-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:29.343656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.343650 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svfw4\" (UniqueName: \"kubernetes.io/projected/3cc809ad-632e-41a5-a022-486056694c47-kube-api-access-svfw4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:29.343656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.343659 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc809ad-632e-41a5-a022-486056694c47-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:29.724491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.724456 5136 generic.go:356] "Generic (PLEG): container finished" podID="3cc809ad-632e-41a5-a022-486056694c47" containerID="2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb" exitCode=0 Nov 02 20:00:29.725022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.724503 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" Nov 02 20:00:29.725022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.724549 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" event={"ID":"3cc809ad-632e-41a5-a022-486056694c47","Type":"ContainerDied","Data":"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb"} Nov 02 20:00:29.725022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.724603 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl" event={"ID":"3cc809ad-632e-41a5-a022-486056694c47","Type":"ContainerDied","Data":"e33e233edeccd1442ca33165195813efd179bc60ff71ef111945450d528055b2"} Nov 02 20:00:29.725022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.724628 5136 scope.go:122] "RemoveContainer" containerID="2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb" Nov 02 20:00:29.734529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.734511 5136 scope.go:122] "RemoveContainer" containerID="2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb" Nov 02 20:00:29.734882 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:29.734861 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb\": container with ID starting with 2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb not found: ID does not exist" containerID="2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb" Nov 02 20:00:29.734944 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.734892 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb"} err="failed to get container status \"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb\": rpc error: code = NotFound desc = could not find container \"2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb\": container with ID starting with 2f81dce4016f41ed8ba51f43cb19378a9772832c2fd834781b7301fe869d8ddb not found: ID does not exist" Nov 02 20:00:29.882537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.882506 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 20:00:29.896950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:29.896887 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d19e8c-848cfdbff-cwhpl"] Nov 02 20:00:30.553754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:30.552718 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" path="/var/lib/kubelet/pods/04a592e0-1d82-45da-a7f1-5b2a3fe3d350/volumes" Nov 02 20:00:30.553754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:30.553332 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc809ad-632e-41a5-a022-486056694c47" path="/var/lib/kubelet/pods/3cc809ad-632e-41a5-a022-486056694c47/volumes" Nov 02 20:00:32.122991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:32.122957 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 20:00:32.123385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:32.123145 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" containerID="cri-o://17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad" gracePeriod=30 Nov 02 20:00:33.449558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.449525 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 20:00:33.449976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.449963 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3cc809ad-632e-41a5-a022-486056694c47" containerName="maas-api" Nov 02 20:00:33.450022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.449976 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc809ad-632e-41a5-a022-486056694c47" containerName="maas-api" Nov 02 20:00:33.450022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.449994 5136 cpu_manager.go:414] "containerMap: removing container" podUID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" containerName="maas-api" Nov 02 20:00:33.450022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.449999 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" containerName="maas-api" Nov 02 20:00:33.450022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450017 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2e71c172-dc08-465c-9689-55d748dc8a60" containerName="maas-api" Nov 02 20:00:33.450022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450021 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e71c172-dc08-465c-9689-55d748dc8a60" containerName="maas-api" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450030 5136 cpu_manager.go:414] "containerMap: removing container" podUID="080710a6-b6c7-4838-9d64-df3ce37b4c00" containerName="payload-pre-processing" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450035 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080710a6-b6c7-4838-9d64-df3ce37b4c00" containerName="payload-pre-processing" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450101 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a592e0-1d82-45da-a7f1-5b2a3fe3d350" containerName="maas-api" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450110 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e71c172-dc08-465c-9689-55d748dc8a60" containerName="maas-api" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450117 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="080710a6-b6c7-4838-9d64-df3ce37b4c00" containerName="payload-pre-processing" Nov 02 20:00:33.450169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.450127 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc809ad-632e-41a5-a022-486056694c47" containerName="maas-api" Nov 02 20:00:33.453519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.453502 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518077 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518185 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518253 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518279 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518307 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518335 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518370 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518415 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.519421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.518792 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.540389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.540349 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a4ab0-openshift-default-dockercfg-xzj7t\"" Nov 02 20:00:33.555953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.555917 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 20:00:33.619497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619366 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619424 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619493 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619572 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619597 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619627 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619654 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619692 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619755 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.619986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.619911 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.620366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.620111 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.620366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.620162 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.620571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.620434 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.620811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.620787 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.623016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.622977 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.625425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.625404 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.674200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.674173 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.674327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.674251 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:33.766083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:33.766006 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:34.179423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.179395 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 20:00:34.180868 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:00:34.180839 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd213e5_e8ba_4aac_b235_5dc90614d08d.slice/crio-7fed349ea7a620f0fb843320aff04a6bee91dba4306ac0c3557cf89cef480c31 WatchSource:0}: Error finding container 7fed349ea7a620f0fb843320aff04a6bee91dba4306ac0c3557cf89cef480c31: Status 404 returned error can't find the container with id 7fed349ea7a620f0fb843320aff04a6bee91dba4306ac0c3557cf89cef480c31 Nov 02 20:00:34.183299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.183264 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:34.183378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.183330 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:34.183378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.183359 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:00:34.755504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.755461 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerStarted","Data":"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997"} Nov 02 20:00:34.755964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.755530 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerStarted","Data":"7fed349ea7a620f0fb843320aff04a6bee91dba4306ac0c3557cf89cef480c31"} Nov 02 20:00:34.766860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.766823 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:34.869302 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:34.869254 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podStartSLOduration=1.869240697 podStartE2EDuration="1.869240697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:34.867892321 +0000 UTC m=+718.474436819" watchObservedRunningTime="2026-11-02 20:00:34.869240697 +0000 UTC m=+718.475785194" Nov 02 20:00:35.709237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:35.709199 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 20:00:35.709490 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:35.709446 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" podUID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" containerName="maas-api" containerID="cri-o://aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd" gracePeriod=30 Nov 02 20:00:35.773247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:35.773215 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:35.978549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:35.978526 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 20:00:36.151026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.150994 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\") pod \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " Nov 02 20:00:36.151216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.151040 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") pod \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " Nov 02 20:00:36.151216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.151070 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\") pod \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " Nov 02 20:00:36.151216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.151121 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\" (UniqueName: \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\") pod \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\" (UID: \"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29\") " Nov 02 20:00:36.151439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.151416 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca" pod "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" (UID: "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:36.153225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.153194 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs" pod "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" (UID: "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:36.153357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.153249 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6" pod "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" (UID: "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29"). InnerVolumeSpecName "kube-api-access-sh2c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:36.153357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.153284 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls" pod "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" (UID: "dca9f0d9-6f8c-42dc-97a5-328ee32c9b29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:36.252319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.252235 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:36.252319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.252265 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:36.252319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.252275 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:36.252319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.252284 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2c6\" (UniqueName: \"kubernetes.io/projected/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29-kube-api-access-sh2c6\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:36.776809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.776766 5136 generic.go:356] "Generic (PLEG): container finished" podID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" containerID="aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd" exitCode=0 Nov 02 20:00:36.777284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.776834 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" Nov 02 20:00:36.777284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.776911 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" event={"ID":"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29","Type":"ContainerDied","Data":"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd"} Nov 02 20:00:36.777284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.776958 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv" event={"ID":"dca9f0d9-6f8c-42dc-97a5-328ee32c9b29","Type":"ContainerDied","Data":"aab1974fd0ebaa165813a1dba8cd6c705e2fcbe7dc08577574b1b11d0e94847b"} Nov 02 20:00:36.777284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.776994 5136 scope.go:122] "RemoveContainer" containerID="aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd" Nov 02 20:00:36.778386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.778082 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:36.779947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.779924 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:00:36.788408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.788389 5136 scope.go:122] "RemoveContainer" containerID="aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd" Nov 02 20:00:36.788677 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:36.788657 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd\": container with ID starting with aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd not found: ID does not exist" containerID="aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd" Nov 02 20:00:36.788790 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:36.788685 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd"} err="failed to get container status \"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd\": rpc error: code = NotFound desc = could not find container \"aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd\": container with ID starting with aeec3332cc76f0b4d589c229af33336eca336158f871e593648c600ec22dc0fd not found: ID does not exist" Nov 02 20:00:37.043933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.043904 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 20:00:37.049746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.049709 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3acf2229-df844d5b9-wkqgv"] Nov 02 20:00:37.373663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.373639 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 20:00:37.466848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466817 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466858 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466885 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466916 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466959 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.466997 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467033 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467062 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467085 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\") pod \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\" (UID: \"1abaf25f-2e4e-4d8d-92db-c6fc52297d51\") " Nov 02 20:00:37.467282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467084 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:37.467480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467313 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:37.467480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467407 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:37.467480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467438 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:37.467685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467501 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.467685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467545 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.467685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467560 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.467685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.467661 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:37.469266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.469246 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "kube-api-access-dvjp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:37.469398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.469279 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:37.469528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.469510 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:37.469581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.469511 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token" pod "1abaf25f-2e4e-4d8d-92db-c6fc52297d51" (UID: "1abaf25f-2e4e-4d8d-92db-c6fc52297d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:37.568472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568439 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvjp9\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-kube-api-access-dvjp9\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.568472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568466 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.568472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568476 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.568708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568485 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.568708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568498 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.568708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.568510 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abaf25f-2e4e-4d8d-92db-c6fc52297d51-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:37.782829 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.782794 5136 generic.go:356] "Generic (PLEG): container finished" podID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerID="17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad" exitCode=0 Nov 02 20:00:37.783232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.782852 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" Nov 02 20:00:37.783232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.782865 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerDied","Data":"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad"} Nov 02 20:00:37.783232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.782921 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc" event={"ID":"1abaf25f-2e4e-4d8d-92db-c6fc52297d51","Type":"ContainerDied","Data":"2f9b7027d26dfee6e4ed05fb7bb2aaac60a9696ab8dfe6ed2358545c2aae1a49"} Nov 02 20:00:37.783232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.782953 5136 scope.go:122] "RemoveContainer" containerID="17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad" Nov 02 20:00:37.791635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.791619 5136 scope.go:122] "RemoveContainer" containerID="3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198" Nov 02 20:00:37.800050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.800035 5136 scope.go:122] "RemoveContainer" containerID="17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad" Nov 02 20:00:37.800307 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:37.800286 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad\": container with ID starting with 17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad not found: ID does not exist" containerID="17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad" Nov 02 20:00:37.800373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.800318 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad"} err="failed to get container status \"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad\": rpc error: code = NotFound desc = could not find container \"17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad\": container with ID starting with 17e37b35121380c5657e7870867ce4c843a8e924a723629858b713b5076760ad not found: ID does not exist" Nov 02 20:00:37.800373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.800344 5136 scope.go:122] "RemoveContainer" containerID="3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198" Nov 02 20:00:37.800616 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:37.800598 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198\": container with ID starting with 3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198 not found: ID does not exist" containerID="3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198" Nov 02 20:00:37.800655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.800622 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198"} err="failed to get container status \"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198\": rpc error: code = NotFound desc = could not find container \"3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198\": container with ID starting with 3c8cabef540a7d5bfeb0f69a2be515749de64addf6405d550708f4fb7bdc5198 not found: ID does not exist" Nov 02 20:00:37.943399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.943366 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 20:00:37.990639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:37.990609 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a0d19e8c-gw-openshift-default-dcb77df7c-79mlc"] Nov 02 20:00:38.545341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:38.545305 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" path="/var/lib/kubelet/pods/1abaf25f-2e4e-4d8d-92db-c6fc52297d51/volumes" Nov 02 20:00:38.545761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:38.545722 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" path="/var/lib/kubelet/pods/dca9f0d9-6f8c-42dc-97a5-328ee32c9b29/volumes" Nov 02 20:00:40.000323 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000290 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 20:00:40.000766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000752 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:40.000808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000767 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:40.000808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000779 5136 cpu_manager.go:414] "containerMap: removing container" podUID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" containerName="maas-api" Nov 02 20:00:40.000808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000785 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" containerName="maas-api" Nov 02 20:00:40.000916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000870 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:40.000916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000884 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:40.000916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.000891 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca9f0d9-6f8c-42dc-97a5-328ee32c9b29" containerName="maas-api" Nov 02 20:00:40.004150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.004127 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.055198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.055163 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-426941\"" Nov 02 20:00:40.093586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.093548 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.093838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.093635 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgl8\" (UniqueName: \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.101970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.101947 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-426941-dockercfg-bt2hh\"" Nov 02 20:00:40.103582 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.103560 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 20:00:40.194982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.194946 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.195161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.194987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgl8\" (UniqueName: \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.195711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.195691 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.249800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.249659 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgl8\" (UniqueName: \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\") pod \"payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.315344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.315268 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.680822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.680793 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 20:00:40.682960 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:00:40.682847 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b019b9f_ba31_4ce5_8b5f_e04e35af4512.slice/crio-37873c8cbc3961d33d910fdee0971d763164f3452c27e2dc5c792506d46db6d4 WatchSource:0}: Error finding container 37873c8cbc3961d33d910fdee0971d763164f3452c27e2dc5c792506d46db6d4: Status 404 returned error can't find the container with id 37873c8cbc3961d33d910fdee0971d763164f3452c27e2dc5c792506d46db6d4 Nov 02 20:00:40.805107 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.805048 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" event={"ID":"2b019b9f-ba31-4ce5-8b5f-e04e35af4512","Type":"ContainerStarted","Data":"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572"} Nov 02 20:00:40.805308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.805136 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" event={"ID":"2b019b9f-ba31-4ce5-8b5f-e04e35af4512","Type":"ContainerStarted","Data":"37873c8cbc3961d33d910fdee0971d763164f3452c27e2dc5c792506d46db6d4"} Nov 02 20:00:40.805308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.805184 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:40.913226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:40.913181 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" podStartSLOduration=1.913167294 podStartE2EDuration="1.913167294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:40.913107825 +0000 UTC m=+724.519652322" watchObservedRunningTime="2026-11-02 20:00:40.913167294 +0000 UTC m=+724.519711791" Nov 02 20:00:45.393875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:45.393836 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 20:00:45.394307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:45.394046 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" containerID="cri-o://5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95" gracePeriod=30 Nov 02 20:00:47.970277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.970207 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 20:00:47.970694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.970681 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:47.970774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.970695 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abaf25f-2e4e-4d8d-92db-c6fc52297d51" containerName="istio-proxy" Nov 02 20:00:47.973238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.973220 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:47.996511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.996488 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtgw\" (UniqueName: \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:47.996598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.996541 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:47.996659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.996635 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:47.996794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:47.996775 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.074790 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.074755 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-426941-serving-cert\"" Nov 02 20:00:48.079334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.079279 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 20:00:48.098273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.098245 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.098425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.098287 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtgw\" (UniqueName: \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.098425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.098327 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.098425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.098358 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.098545 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:48.098462 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-serving-cert: secret "maas-api-e2e-worker-w2-426941-serving-cert" not found Nov 02 20:00:48.098545 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:48.098543 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls podName:d072c3af-98c3-4a14-9810-c1385806e003 nodeName:}" failed. No retries permitted until 2026-11-02 20:00:48.598522795 +0000 UTC m=+732.205067288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls") pod "maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" (UID: "d072c3af-98c3-4a14-9810-c1385806e003") : secret "maas-api-e2e-worker-w2-426941-serving-cert" not found Nov 02 20:00:48.098887 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.098866 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.100773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.100754 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.154163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.154131 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtgw\" (UniqueName: \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.603498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.603456 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.606074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.606046 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") pod \"maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:48.886018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:48.885942 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:49.249071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:49.249047 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 20:00:49.251011 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:00:49.250978 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd072c3af_98c3_4a14_9810_c1385806e003.slice/crio-1daadee9006b014e7eced0efe84fde18f63b7882d3783590b8e4f11848ce114e WatchSource:0}: Error finding container 1daadee9006b014e7eced0efe84fde18f63b7882d3783590b8e4f11848ce114e: Status 404 returned error can't find the container with id 1daadee9006b014e7eced0efe84fde18f63b7882d3783590b8e4f11848ce114e Nov 02 20:00:49.857038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:49.857004 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" event={"ID":"d072c3af-98c3-4a14-9810-c1385806e003","Type":"ContainerStarted","Data":"1daadee9006b014e7eced0efe84fde18f63b7882d3783590b8e4f11848ce114e"} Nov 02 20:00:50.680000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.679929 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:50.724595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724565 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724619 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724646 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724770 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724871 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724916 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724936 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:50.724993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724949 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.724993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.724978 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.725186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725045 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\") pod \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\" (UID: \"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1\") " Nov 02 20:00:50.725233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725188 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:50.725370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725219 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:50.725370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725325 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:50.725495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725414 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:50.725541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725487 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.725541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725509 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.725541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725525 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.725648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.725539 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.726914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.726890 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:50.726914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.726910 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:50.727209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.727192 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "kube-api-access-7lhmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:50.727268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.727239 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token" pod "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" (UID: "3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:50.838384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.827465 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.838384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.827497 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.838384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.827513 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.838384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.827613 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.838384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.827632 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhmd\" (UniqueName: \"kubernetes.io/projected/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1-kube-api-access-7lhmd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:50.864285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.864252 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" event={"ID":"d072c3af-98c3-4a14-9810-c1385806e003","Type":"ContainerStarted","Data":"c1b81c27b1e9c64355b38fbe3682c7c573c3cb0d3a2aec59dd49fe44ee10576e"} Nov 02 20:00:50.864463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.864364 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:50.866196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.866170 5136 generic.go:356] "Generic (PLEG): container finished" podID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerID="5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95" exitCode=0 Nov 02 20:00:50.866339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.866211 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" Nov 02 20:00:50.866339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.866252 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerDied","Data":"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95"} Nov 02 20:00:50.866339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.866312 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4" event={"ID":"3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1","Type":"ContainerDied","Data":"0a3d780b5fd0ca25c8af0ff33d7ec29e9f8d8066cedb5f1410e0819682d68262"} Nov 02 20:00:50.866499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.866343 5136 scope.go:122] "RemoveContainer" containerID="5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95" Nov 02 20:00:50.877087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.877064 5136 scope.go:122] "RemoveContainer" containerID="f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2" Nov 02 20:00:50.886646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.886625 5136 scope.go:122] "RemoveContainer" containerID="5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95" Nov 02 20:00:50.887283 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:50.887253 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95\": container with ID starting with 5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95 not found: ID does not exist" containerID="5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95" Nov 02 20:00:50.887370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.887295 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95"} err="failed to get container status \"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95\": rpc error: code = NotFound desc = could not find container \"5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95\": container with ID starting with 5205fbdfa3d35d6153c921d079d3f43839ccd9cd46272dccb0366c5eb33bbd95 not found: ID does not exist" Nov 02 20:00:50.887370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.887319 5136 scope.go:122] "RemoveContainer" containerID="f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2" Nov 02 20:00:50.887590 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:50.887570 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2\": container with ID starting with f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2 not found: ID does not exist" containerID="f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2" Nov 02 20:00:50.887639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.887596 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2"} err="failed to get container status \"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2\": rpc error: code = NotFound desc = could not find container \"f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2\": container with ID starting with f077defd70b0f96c5cd41ac283e05c85a43161caa1ff3aa6d4daa51dd33c7cd2 not found: ID does not exist" Nov 02 20:00:50.973480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:50.973429 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" podStartSLOduration=3.10384793 podStartE2EDuration="3.973415589s" totalImagesPullingTime="869.567659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:50.972764061 +0000 UTC m=+734.579308559" watchObservedRunningTime="2026-11-02 20:00:50.973415589 +0000 UTC m=+734.579960088" Nov 02 20:00:51.005457 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:51.005420 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:00:51.080044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:51.080016 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 20:00:51.084472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:51.084447 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3acf2229-openshift-default-9b55876cc-qqnt4"] Nov 02 20:00:51.815520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:51.815492 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 20:00:52.545766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:52.545719 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" path="/var/lib/kubelet/pods/3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1/volumes" Nov 02 20:00:53.401475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.401445 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 20:00:53.401939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.401923 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.401983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.401940 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.401983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.401971 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.401983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.401976 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.402078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.402034 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.402078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.402044 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f74ef6e-6b06-4701-b0ce-2cbc6c4552f1" containerName="istio-proxy" Nov 02 20:00:53.405225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.405205 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.457319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.457293 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-14d090-serving-cert\"" Nov 02 20:00:53.466125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.466096 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.466283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.466234 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.466356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.466286 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ngk\" (UniqueName: \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.466356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.466332 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.506537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.506512 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 20:00:53.567684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.567649 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.567857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.567719 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.567857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.567766 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ngk\" (UniqueName: \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.567857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.567801 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.567857 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:53.567846 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-serving-cert: secret "maas-api-e2e-worker-w0-14d090-serving-cert" not found Nov 02 20:00:53.568073 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:53.567934 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls podName:b80e468e-ea96-4557-9c0b-9ae84f6895e1 nodeName:}" failed. No retries permitted until 2026-11-02 20:00:54.067911926 +0000 UTC m=+737.674456406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls") pod "maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" (UID: "b80e468e-ea96-4557-9c0b-9ae84f6895e1") : secret "maas-api-e2e-worker-w0-14d090-serving-cert" not found Nov 02 20:00:53.568601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.568573 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.570326 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.570298 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:53.624339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:53.624316 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ngk\" (UniqueName: \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:54.073218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:54.073187 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:54.075531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:54.075512 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:54.318342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:54.318312 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:54.682533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:54.682510 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 20:00:54.684407 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:00:54.684382 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80e468e_ea96_4557_9c0b_9ae84f6895e1.slice/crio-3c58762c685b6e211ae08437bac85d7ef00d9b96e1ff10b8a10651ab1b42d5de WatchSource:0}: Error finding container 3c58762c685b6e211ae08437bac85d7ef00d9b96e1ff10b8a10651ab1b42d5de: Status 404 returned error can't find the container with id 3c58762c685b6e211ae08437bac85d7ef00d9b96e1ff10b8a10651ab1b42d5de Nov 02 20:00:54.890780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:54.890747 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" event={"ID":"b80e468e-ea96-4557-9c0b-9ae84f6895e1","Type":"ContainerStarted","Data":"3c58762c685b6e211ae08437bac85d7ef00d9b96e1ff10b8a10651ab1b42d5de"} Nov 02 20:00:55.899060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:55.898966 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" event={"ID":"b80e468e-ea96-4557-9c0b-9ae84f6895e1","Type":"ContainerStarted","Data":"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751"} Nov 02 20:00:55.899444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:55.899105 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:00:56.011617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:56.011567 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" podStartSLOduration=2.17949185 podStartE2EDuration="3.011553347s" totalImagesPullingTime="832.061497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:56.009855267 +0000 UTC m=+739.616399764" watchObservedRunningTime="2026-11-02 20:00:56.011553347 +0000 UTC m=+739.618097859" Nov 02 20:00:56.877281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:56.877254 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 20:00:58.640469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.640435 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 20:00:58.643240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.643218 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.722576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.722537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.722763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.722583 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.722829 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.722805 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwt7n\" (UniqueName: \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.722867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.722838 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.742304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.742277 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-6a4ab0-serving-cert\"" Nov 02 20:00:58.744049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.744028 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 20:00:58.823711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.823685 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwt7n\" (UniqueName: \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.823867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.823717 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.823867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.823788 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.823867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.823811 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.823995 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:58.823974 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-serving-cert: secret "maas-api-e2e-worker-w3-6a4ab0-serving-cert" not found Nov 02 20:00:58.824060 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:00:58.824049 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls podName:54fd3124-1e2d-443d-82c1-70084f1e299d nodeName:}" failed. No retries permitted until 2026-11-02 20:00:59.324030863 +0000 UTC m=+742.930575350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls") pod "maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" (UID: "54fd3124-1e2d-443d-82c1-70084f1e299d") : secret "maas-api-e2e-worker-w3-6a4ab0-serving-cert" not found Nov 02 20:00:58.824375 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.824350 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.826207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.826182 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:58.879785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:58.879759 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwt7n\" (UniqueName: \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:59.337874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:59.337839 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:59.340335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:59.340307 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:59.556063 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:59.556027 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:00:59.945974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:00:59.945951 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 20:00:59.947260 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:00:59.947226 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fd3124_1e2d_443d_82c1_70084f1e299d.slice/crio-c87f5beed816ce561d5deed7b26b337ccdb17981fec24b5079144aad1973d99e WatchSource:0}: Error finding container c87f5beed816ce561d5deed7b26b337ccdb17981fec24b5079144aad1973d99e: Status 404 returned error can't find the container with id c87f5beed816ce561d5deed7b26b337ccdb17981fec24b5079144aad1973d99e Nov 02 20:01:00.934412 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:00.934370 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" event={"ID":"54fd3124-1e2d-443d-82c1-70084f1e299d","Type":"ContainerStarted","Data":"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc"} Nov 02 20:01:00.934579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:00.934430 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" event={"ID":"54fd3124-1e2d-443d-82c1-70084f1e299d","Type":"ContainerStarted","Data":"c87f5beed816ce561d5deed7b26b337ccdb17981fec24b5079144aad1973d99e"} Nov 02 20:01:00.934579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:00.934484 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:01:01.048770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:01.048650 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" podStartSLOduration=2.242778526 podStartE2EDuration="3.048634864s" totalImagesPullingTime="805.856338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:01.046933582 +0000 UTC m=+744.653478081" watchObservedRunningTime="2026-11-02 20:01:01.048634864 +0000 UTC m=+744.655179362" Nov 02 20:01:01.909466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:01.909429 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 20:01:03.183160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.183125 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 20:01:03.183622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.183318 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" podUID="dfc1a814-aa10-4c18-9097-de7c82467610" containerName="maas-api" containerID="cri-o://509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39" gracePeriod=30 Nov 02 20:01:03.454087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.454059 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 20:01:03.584114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.584069 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\" (UniqueName: \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\") pod \"dfc1a814-aa10-4c18-9097-de7c82467610\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " Nov 02 20:01:03.584324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.584149 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") pod \"dfc1a814-aa10-4c18-9097-de7c82467610\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " Nov 02 20:01:03.584324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.584269 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\") pod \"dfc1a814-aa10-4c18-9097-de7c82467610\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " Nov 02 20:01:03.584450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.584352 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\") pod \"dfc1a814-aa10-4c18-9097-de7c82467610\" (UID: \"dfc1a814-aa10-4c18-9097-de7c82467610\") " Nov 02 20:01:03.586697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.586640 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca" pod "dfc1a814-aa10-4c18-9097-de7c82467610" (UID: "dfc1a814-aa10-4c18-9097-de7c82467610"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:03.587379 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.587357 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1a814-aa10-4c18-9097-de7c82467610-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:03.589242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.589216 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq" pod "dfc1a814-aa10-4c18-9097-de7c82467610" (UID: "dfc1a814-aa10-4c18-9097-de7c82467610"). InnerVolumeSpecName "kube-api-access-fbxjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:03.589689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.589665 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls" pod "dfc1a814-aa10-4c18-9097-de7c82467610" (UID: "dfc1a814-aa10-4c18-9097-de7c82467610"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:03.590363 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.590327 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs" pod "dfc1a814-aa10-4c18-9097-de7c82467610" (UID: "dfc1a814-aa10-4c18-9097-de7c82467610"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:03.694814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.692707 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:03.694814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.692820 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxjq\" (UniqueName: \"kubernetes.io/projected/dfc1a814-aa10-4c18-9097-de7c82467610-kube-api-access-fbxjq\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:03.694814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.692839 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc1a814-aa10-4c18-9097-de7c82467610-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:03.953189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.953149 5136 generic.go:356] "Generic (PLEG): container finished" podID="dfc1a814-aa10-4c18-9097-de7c82467610" containerID="509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39" exitCode=0 Nov 02 20:01:03.953390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.953206 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" Nov 02 20:01:03.953390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.953234 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" event={"ID":"dfc1a814-aa10-4c18-9097-de7c82467610","Type":"ContainerDied","Data":"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39"} Nov 02 20:01:03.953390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.953294 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj" event={"ID":"dfc1a814-aa10-4c18-9097-de7c82467610","Type":"ContainerDied","Data":"87b61a35102cf28ef4379381f4df8c2ec45dd43d4e75406b31d6716f52bc5569"} Nov 02 20:01:03.953390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.953318 5136 scope.go:122] "RemoveContainer" containerID="509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39" Nov 02 20:01:03.964632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.964608 5136 scope.go:122] "RemoveContainer" containerID="509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39" Nov 02 20:01:03.964948 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:03.964926 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39\": container with ID starting with 509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39 not found: ID does not exist" containerID="509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39" Nov 02 20:01:03.965026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:03.964965 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39"} err="failed to get container status \"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39\": rpc error: code = NotFound desc = could not find container \"509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39\": container with ID starting with 509204597e8676c634b44a0534b3dfad01aa647758649ba810773bb6a2fc9c39 not found: ID does not exist" Nov 02 20:01:04.108628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:04.108591 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 20:01:04.156195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:04.156158 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-172adb07-dcbd76df8-zhnpj"] Nov 02 20:01:04.545376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:04.545344 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc1a814-aa10-4c18-9097-de7c82467610" path="/var/lib/kubelet/pods/dfc1a814-aa10-4c18-9097-de7c82467610/volumes" Nov 02 20:01:05.500234 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:05.500199 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 20:01:05.500470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:05.500441 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" containerID="cri-o://57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb" gracePeriod=30 Nov 02 20:01:06.867392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.867359 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 20:01:06.867847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.867833 5136 cpu_manager.go:414] "containerMap: removing container" podUID="dfc1a814-aa10-4c18-9097-de7c82467610" containerName="maas-api" Nov 02 20:01:06.867903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.867848 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc1a814-aa10-4c18-9097-de7c82467610" containerName="maas-api" Nov 02 20:01:06.867945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.867935 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc1a814-aa10-4c18-9097-de7c82467610" containerName="maas-api" Nov 02 20:01:06.871395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.871375 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:06.921380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.921339 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\"" Nov 02 20:01:06.970230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.970203 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 20:01:06.971535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:06.971512 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 20:01:07.065921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.065885 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.065921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.065920 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq9q\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.065947 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066043 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066085 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066103 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066246 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066370 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.066475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.066462 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167032 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167079 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167137 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167169 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167248 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167338 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167366 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq9q\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167416 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167436 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167446 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167618 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167898 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.167982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.167953 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.168182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.168161 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.170314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.170290 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.170535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.170515 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.221652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.221620 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.221821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.221798 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq9q\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.483974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.483945 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:07.895292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.895263 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 20:01:07.896463 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:01:07.896409 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc4f3c8_0d3d_4fd9_a44e_9e6c90461c01.slice/crio-a1d76b801ee70fc81d59db03ce0c990af0089b5ecb9f5db91fbaab38ee66c444 WatchSource:0}: Error finding container a1d76b801ee70fc81d59db03ce0c990af0089b5ecb9f5db91fbaab38ee66c444: Status 404 returned error can't find the container with id a1d76b801ee70fc81d59db03ce0c990af0089b5ecb9f5db91fbaab38ee66c444 Nov 02 20:01:07.899121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.899081 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:07.899208 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.899168 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:07.899246 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.899210 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:07.979120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.979091 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerStarted","Data":"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457"} Nov 02 20:01:07.979245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:07.979150 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerStarted","Data":"a1d76b801ee70fc81d59db03ce0c990af0089b5ecb9f5db91fbaab38ee66c444"} Nov 02 20:01:08.091086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:08.091039 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podStartSLOduration=2.091026545 podStartE2EDuration="2.091026545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:08.090349727 +0000 UTC m=+751.696894226" watchObservedRunningTime="2026-11-02 20:01:08.091026545 +0000 UTC m=+751.697571045" Nov 02 20:01:08.484914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:08.484880 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:08.490140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:08.490110 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:08.984419 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:08.984387 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:08.985572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:08.985550 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:10.759809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.759782 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:01:10.798188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798086 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798137 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798215 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798262 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798292 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798319 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798392 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798423 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.798774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798668 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:10.798955 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798817 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.798955 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.798878 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:10.799138 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.799112 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:10.799239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.799183 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:10.800473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.800443 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:10.801205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.801181 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:10.801329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.801302 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:10.801383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.801305 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "kube-api-access-vmhhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:10.899367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899335 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\") pod \"64c51300-cb4c-4584-a29e-9fcdcea9e916\" (UID: \"64c51300-cb4c-4584-a29e-9fcdcea9e916\") " Nov 02 20:01:10.899586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899572 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhhq\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-kube-api-access-vmhhq\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899589 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899598 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c51300-cb4c-4584-a29e-9fcdcea9e916-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899608 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899618 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899631 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.899806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.899640 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c51300-cb4c-4584-a29e-9fcdcea9e916-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:10.901354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.901333 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo" pod "64c51300-cb4c-4584-a29e-9fcdcea9e916" (UID: "64c51300-cb4c-4584-a29e-9fcdcea9e916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:01:10.995285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.995245 5136 generic.go:356] "Generic (PLEG): container finished" podID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerID="57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb" exitCode=0 Nov 02 20:01:10.995452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.995286 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerDied","Data":"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb"} Nov 02 20:01:10.995452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.995343 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" event={"ID":"64c51300-cb4c-4584-a29e-9fcdcea9e916","Type":"ContainerDied","Data":"a5243a183b02ea2a7e5278c3e65a6c1c1ac2dd54c69635f80470ce95a5974f44"} Nov 02 20:01:10.995452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.995368 5136 scope.go:122] "RemoveContainer" containerID="57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb" Nov 02 20:01:10.995452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:10.995295 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp" Nov 02 20:01:11.000149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.000120 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c51300-cb4c-4584-a29e-9fcdcea9e916-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:11.004985 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.004972 5136 scope.go:122] "RemoveContainer" containerID="f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0" Nov 02 20:01:11.013037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.013014 5136 scope.go:122] "RemoveContainer" containerID="57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb" Nov 02 20:01:11.013297 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:11.013279 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb\": container with ID starting with 57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb not found: ID does not exist" containerID="57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb" Nov 02 20:01:11.013383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.013310 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb"} err="failed to get container status \"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb\": rpc error: code = NotFound desc = could not find container \"57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb\": container with ID starting with 57895dbd59fae7110e68e75333ca73e7fa513541c6743e8b24d8dd5f6b22b5fb not found: ID does not exist" Nov 02 20:01:11.013383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.013335 5136 scope.go:122] "RemoveContainer" containerID="f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0" Nov 02 20:01:11.013587 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:11.013570 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0\": container with ID starting with f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0 not found: ID does not exist" containerID="f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0" Nov 02 20:01:11.013623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.013596 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0"} err="failed to get container status \"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0\": rpc error: code = NotFound desc = could not find container \"f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0\": container with ID starting with f0fbfd366b54d7cf09ceadb98054a76c599dc55013724b5007a4c48831aacbd0 not found: ID does not exist" Nov 02 20:01:11.154371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.154292 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 20:01:11.158311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:11.158285 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-172adb07-openshift-default-665cbb8ccc-9trpp"] Nov 02 20:01:12.550631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:12.550593 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" path="/var/lib/kubelet/pods/64c51300-cb4c-4584-a29e-9fcdcea9e916/volumes" Nov 02 20:01:14.660788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.660751 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 20:01:14.661406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661385 5136 cpu_manager.go:414] "containerMap: removing container" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.661519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661406 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.661519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661423 5136 cpu_manager.go:414] "containerMap: removing container" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.661519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661431 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.661675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661555 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.661675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.661569 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c51300-cb4c-4584-a29e-9fcdcea9e916" containerName="istio-proxy" Nov 02 20:01:14.665039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.665016 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.715013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.714975 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-bbee67\"" Nov 02 20:01:14.737959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.737930 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.738158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.737987 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4x5\" (UniqueName: \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.762487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.762461 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-bbee67-dockercfg-xscq4\"" Nov 02 20:01:14.764264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.764242 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 20:01:14.839264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.839225 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4x5\" (UniqueName: \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.839433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.839369 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.840078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.840052 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.919415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.919336 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4x5\" (UniqueName: \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\") pod \"payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:14.979620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:14.979591 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:15.384010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.383982 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 20:01:15.385239 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:01:15.385215 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6f95a6_eee7_4b3e_8aff_fb99a6214377.slice/crio-8602d5927bb2484edaff185c64dc7eeaafcf5bcd712f6f967c55125699ac4deb WatchSource:0}: Error finding container 8602d5927bb2484edaff185c64dc7eeaafcf5bcd712f6f967c55125699ac4deb: Status 404 returned error can't find the container with id 8602d5927bb2484edaff185c64dc7eeaafcf5bcd712f6f967c55125699ac4deb Nov 02 20:01:15.649314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.649232 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 20:01:15.652192 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.652170 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.709745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.708969 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 20:01:15.709745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.709691 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c25349-serving-cert\"" Nov 02 20:01:15.748910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.748882 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.749091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.748939 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.749091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.749019 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczn9\" (UniqueName: \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.749185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.749115 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.849657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.849630 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.849870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.849685 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.849870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.849704 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczn9\" (UniqueName: \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.849870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.849762 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.849870 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:15.849799 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-serving-cert: secret "maas-api-e2e-worker-w4-c25349-serving-cert" not found Nov 02 20:01:15.850092 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:15.849887 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls podName:f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f nodeName:}" failed. No retries permitted until 2026-11-02 20:01:16.349864532 +0000 UTC m=+759.956409026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls") pod "maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" (UID: "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f") : secret "maas-api-e2e-worker-w4-c25349-serving-cert" not found Nov 02 20:01:15.850413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.850386 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.852040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.852024 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:15.916336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:15.916249 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczn9\" (UniqueName: \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:16.029236 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.029198 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" event={"ID":"7b6f95a6-eee7-4b3e-8aff-fb99a6214377","Type":"ContainerStarted","Data":"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075"} Nov 02 20:01:16.029432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.029273 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" event={"ID":"7b6f95a6-eee7-4b3e-8aff-fb99a6214377","Type":"ContainerStarted","Data":"8602d5927bb2484edaff185c64dc7eeaafcf5bcd712f6f967c55125699ac4deb"} Nov 02 20:01:16.029432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.029321 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:16.142842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.142782 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" podStartSLOduration=2.142757826 podStartE2EDuration="2.142757826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:16.141966724 +0000 UTC m=+759.748511224" watchObservedRunningTime="2026-11-02 20:01:16.142757826 +0000 UTC m=+759.749302327" Nov 02 20:01:16.352841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.352801 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:16.355255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.355230 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:16.564449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:16.564417 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:17.015027 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:01:17.014992 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1297967_4c0f_4520_b2f9_fdbcfe3c5d5f.slice/crio-8f1cf9631b6e8a0d9d999afc9a0420b65576e383080996d140e48eed78948012 WatchSource:0}: Error finding container 8f1cf9631b6e8a0d9d999afc9a0420b65576e383080996d140e48eed78948012: Status 404 returned error can't find the container with id 8f1cf9631b6e8a0d9d999afc9a0420b65576e383080996d140e48eed78948012 Nov 02 20:01:17.015421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:17.015170 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 20:01:17.035049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:17.035021 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" event={"ID":"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f","Type":"ContainerStarted","Data":"8f1cf9631b6e8a0d9d999afc9a0420b65576e383080996d140e48eed78948012"} Nov 02 20:01:18.047238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:18.047192 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" event={"ID":"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f","Type":"ContainerStarted","Data":"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665"} Nov 02 20:01:18.047684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:18.047275 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:18.175949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:18.175844 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" podStartSLOduration=2.315086518 podStartE2EDuration="3.17582989s" totalImagesPullingTime="860.743372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:18.174573755 +0000 UTC m=+761.781118253" watchObservedRunningTime="2026-11-02 20:01:18.17582989 +0000 UTC m=+761.782374408" Nov 02 20:01:21.005939 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:21.005901 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:01:22.693593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:22.693569 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:43602->10.133.0.51:15021: read: connection reset by peer" start-of-body= Nov 02 20:01:22.694048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:22.693634 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:43602->10.133.0.51:15021: read: connection reset by peer" Nov 02 20:01:23.074529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.074497 5136 generic.go:356] "Generic (PLEG): container finished" podID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerID="5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997" exitCode=0 Nov 02 20:01:23.074529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.074534 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerDied","Data":"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997"} Nov 02 20:01:23.074932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.074914 5136 scope.go:122] "RemoveContainer" containerID="5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997" Nov 02 20:01:23.076081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.076048 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:23.076191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.076116 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:23.076191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:23.076156 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:24.058590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:24.058562 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 20:01:24.083068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:24.083040 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerStarted","Data":"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f"} Nov 02 20:01:24.766592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:24.766557 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:01:24.766804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:24.766702 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:01:24.766804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:24.766778 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 20:01:25.766932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:25.766892 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:01:25.767418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:25.766964 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 20:01:26.766593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:26.766561 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:01:26.766788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:26.766609 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 20:01:27.038987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:27.038915 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 20:01:27.766393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:27.766362 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:01:27.766564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:27.766408 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 20:01:28.766817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:28.766786 5136 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:01:28.767210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:28.766831 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 20:01:29.770387 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:29.770359 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:01:29.772755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:29.770626 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:01:29.776599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:29.775618 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 20:01:37.849768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.849720 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 20:01:37.853952 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.853932 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:37.951488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.951451 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-bbee67-serving-cert\"" Nov 02 20:01:37.953241 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.953220 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 20:01:37.978070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.978042 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:37.978182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.978095 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:37.978236 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.978219 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq54d\" (UniqueName: \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:37.978277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:37.978265 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.079759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.079712 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.079924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.079795 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq54d\" (UniqueName: \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.079924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.079842 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.079924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.079900 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.080094 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:38.079999 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-serving-cert: secret "maas-api-e2e-worker-w1-bbee67-serving-cert" not found Nov 02 20:01:38.080094 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:38.080055 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls podName:1d758097-e052-4a9e-b5ee-4ee30c33b38f nodeName:}" failed. No retries permitted until 2026-11-02 20:01:38.580034732 +0000 UTC m=+782.186579211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls") pod "maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" (UID: "1d758097-e052-4a9e-b5ee-4ee30c33b38f") : secret "maas-api-e2e-worker-w1-bbee67-serving-cert" not found Nov 02 20:01:38.080481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.080458 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.082390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.082364 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.135117 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.135046 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq54d\" (UniqueName: \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.583541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.583506 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.585943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.585920 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:38.766278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:38.766243 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:39.140107 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:39.140070 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 20:01:39.141140 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:01:39.141110 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d758097_e052_4a9e_b5ee_4ee30c33b38f.slice/crio-119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102 WatchSource:0}: Error finding container 119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102: Status 404 returned error can't find the container with id 119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102 Nov 02 20:01:39.187762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:39.187714 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" event={"ID":"1d758097-e052-4a9e-b5ee-4ee30c33b38f","Type":"ContainerStarted","Data":"119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102"} Nov 02 20:01:40.194119 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:40.194078 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" event={"ID":"1d758097-e052-4a9e-b5ee-4ee30c33b38f","Type":"ContainerStarted","Data":"9ccb45fb91c7a3a01c3f8d820e077bcf109fb86531e214a192717c8da0c2ec7d"} Nov 02 20:01:40.194553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:40.194239 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:40.305047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:40.304958 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" podStartSLOduration=2.452505985 podStartE2EDuration="3.304944511s" totalImagesPullingTime="852.438526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:40.303418168 +0000 UTC m=+783.909962666" watchObservedRunningTime="2026-11-02 20:01:40.304944511 +0000 UTC m=+783.911489009" Nov 02 20:01:46.216757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:46.216715 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 20:01:51.002693 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:01:51.002653 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:01:53.986098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:53.986021 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:53.986098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:53.986077 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:01:54.264384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.264347 5136 generic.go:356] "Generic (PLEG): container finished" podID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerID="b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457" exitCode=0 Nov 02 20:01:54.264552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.264404 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerDied","Data":"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457"} Nov 02 20:01:54.264808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.264794 5136 scope.go:122] "RemoveContainer" containerID="b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457" Nov 02 20:01:54.266088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.266060 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:54.266166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.266129 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:54.266166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:54.266161 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:01:55.271445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:55.271410 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerStarted","Data":"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f"} Nov 02 20:01:55.484294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:55.484259 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:01:55.484469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:55.484416 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:55.484469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:55.484459 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:01:56.485269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:56.485238 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:56.485656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:56.485284 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:01:57.484465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:57.484434 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:57.484645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:57.484483 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:01:58.484630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:58.484597 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:58.485022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:58.484652 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:01:59.485183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:59.485147 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:01:59.485599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:01:59.485209 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:02:00.485205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:02:00.485173 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:02:00.485702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:02:00.485226 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 20:02:01.489078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:02:01.489051 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:02:01.489489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:02:01.489302 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:02:01.490046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:02:01.490028 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 20:02:21.002880 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:02:21.002805 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:02:51.004014 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:02:51.003975 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:03:21.003744 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:03:21.003689 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:03:31.282427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.282390 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 20:03:31.286036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.286000 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.360222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.360191 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 20:03:31.375216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375189 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.375358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375222 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.375358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375250 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btj6\" (UniqueName: \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.375452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375395 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.375452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375432 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.375542 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.375527 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.384024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.384005 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"openshift-service-ca.crt\"" Nov 02 20:03:31.384124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.384027 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"default-dockercfg-hn4wb\"" Nov 02 20:03:31.384124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.384031 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"kube-root-ca.crt\"" Nov 02 20:03:31.384124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.384047 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"premium-w4-c25349-kserve-self-signed-certs\"" Nov 02 20:03:31.476556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476525 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.476556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476561 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.476836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476628 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.476836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476649 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.476836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476667 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.476836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.476775 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btj6\" (UniqueName: \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.477060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.477000 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.477060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.477022 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.477157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.477085 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.479078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.479050 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.479230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.479212 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.530452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.530424 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btj6\" (UniqueName: \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\") pod \"premium-w4-c25349-kserve-66f686f5b4-b5h7q\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.597588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.597524 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:31.998917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:31.998888 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 20:03:32.000090 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:03:32.000066 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42f903d4_b68e_4291_869f_2092f360a459.slice/crio-41563519e6da740c0d957d8bd9ae6ea98b8cd617c8ce8e9bbebb992d28f5583f WatchSource:0}: Error finding container 41563519e6da740c0d957d8bd9ae6ea98b8cd617c8ce8e9bbebb992d28f5583f: Status 404 returned error can't find the container with id 41563519e6da740c0d957d8bd9ae6ea98b8cd617c8ce8e9bbebb992d28f5583f Nov 02 20:03:32.800535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:32.800501 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" event={"ID":"42f903d4-b68e-4291-869f-2092f360a459","Type":"ContainerStarted","Data":"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387"} Nov 02 20:03:32.800535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:32.800546 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" event={"ID":"42f903d4-b68e-4291-869f-2092f360a459","Type":"ContainerStarted","Data":"41563519e6da740c0d957d8bd9ae6ea98b8cd617c8ce8e9bbebb992d28f5583f"} Nov 02 20:03:32.910549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:32.910500 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podStartSLOduration=1.910487134 podStartE2EDuration="1.910487134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:32.909122702 +0000 UTC m=+896.515667200" watchObservedRunningTime="2026-11-02 20:03:32.910487134 +0000 UTC m=+896.517031632" Nov 02 20:03:33.598357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:33.598322 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:39.305376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:39.303866 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 20:03:39.305376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:39.304436 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" containerID="cri-o://5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b" gracePeriod=30 Nov 02 20:03:41.598055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:41.598022 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:41.610802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:41.610776 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:41.857608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:41.857533 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:03:44.565261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.565238 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 20:03:44.613037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.612971 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613005 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613107 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613139 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613174 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613230 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613261 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613268 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.613513 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613422 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613509 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\") pod \"53758626-0b3c-4e81-9809-b8df32c09757\" (UID: \"53758626-0b3c-4e81-9809-b8df32c09757\") " Nov 02 20:03:44.613701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613475 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.613701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613547 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.613701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613558 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.613964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613830 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:44.613964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613936 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.613964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613955 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.613964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613964 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53758626-0b3c-4e81-9809-b8df32c09757-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.614183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613973 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.614183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.613982 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.615492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.615466 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "kube-api-access-cv6ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:44.615587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.615491 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:44.615807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.615787 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:03:44.615872 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.615847 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy" pod "53758626-0b3c-4e81-9809-b8df32c09757" (UID: "53758626-0b3c-4e81-9809-b8df32c09757"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.715571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.715531 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6ck\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-kube-api-access-cv6ck\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.715571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.715570 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53758626-0b3c-4e81-9809-b8df32c09757-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.715802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.715586 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53758626-0b3c-4e81-9809-b8df32c09757-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.715802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.715600 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53758626-0b3c-4e81-9809-b8df32c09757-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.864663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.864576 5136 generic.go:356] "Generic (PLEG): container finished" podID="53758626-0b3c-4e81-9809-b8df32c09757" containerID="5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b" exitCode=0 Nov 02 20:03:44.864663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.864624 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" event={"ID":"53758626-0b3c-4e81-9809-b8df32c09757","Type":"ContainerDied","Data":"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b"} Nov 02 20:03:44.864663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.864628 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" Nov 02 20:03:44.864663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.864665 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp" event={"ID":"53758626-0b3c-4e81-9809-b8df32c09757","Type":"ContainerDied","Data":"e77882c470d849e0a3814f4aee467163ca420826ecf248ed344e4eea60c1ef57"} Nov 02 20:03:44.864956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.864685 5136 scope.go:122] "RemoveContainer" containerID="5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b" Nov 02 20:03:44.882829 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.882805 5136 scope.go:122] "RemoveContainer" containerID="5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b" Nov 02 20:03:44.883148 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:03:44.883118 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b\": container with ID starting with 5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b not found: ID does not exist" containerID="5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b" Nov 02 20:03:44.883243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:44.883158 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b"} err="failed to get container status \"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b\": rpc error: code = NotFound desc = could not find container \"5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b\": container with ID starting with 5f9961f4e029bec19ea3364071c4012ff907c2250eb7add41dda84e51426856b not found: ID does not exist" Nov 02 20:03:45.020064 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:45.020028 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 20:03:45.024335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:45.024304 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2hpkp"] Nov 02 20:03:46.547662 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:03:46.547589 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53758626-0b3c-4e81-9809-b8df32c09757" path="/var/lib/kubelet/pods/53758626-0b3c-4e81-9809-b8df32c09757/volumes" Nov 02 20:03:51.003352 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:03:51.003319 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:04:01.624535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.624501 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:04:01.625034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.624993 5136 cpu_manager.go:414] "containerMap: removing container" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" Nov 02 20:04:01.625034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.625005 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" Nov 02 20:04:01.625109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.625080 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="53758626-0b3c-4e81-9809-b8df32c09757" containerName="istio-proxy" Nov 02 20:04:01.628394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.628375 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.678404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.678382 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-426941-openshift-default-dockercfg-mxlvf\"" Nov 02 20:04:01.728060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.728033 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:04:01.777622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777587 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777628 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777659 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777706 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777832 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777901 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgndz\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.777977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777951 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.778119 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.777995 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.778119 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.778011 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879350 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879394 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879565 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879592 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879621 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879760 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgndz\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879807 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879815 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879768 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879889 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.879920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879914 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.880152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.879939 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.880190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.880171 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.880340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.880317 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.882033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.882009 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.882268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.882250 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.934265 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.934237 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.934423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.934339 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgndz\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:01.941011 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:01.940988 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:02.362062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.362034 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:04:02.363101 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:04:02.363073 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf5f1b4_3105_4a64_b23c_23fe7bf88d51.slice/crio-6e5ddb6c2ed796ce3746f369a92fcfa72218a88d07257aedb38a376c35bc4bba WatchSource:0}: Error finding container 6e5ddb6c2ed796ce3746f369a92fcfa72218a88d07257aedb38a376c35bc4bba: Status 404 returned error can't find the container with id 6e5ddb6c2ed796ce3746f369a92fcfa72218a88d07257aedb38a376c35bc4bba Nov 02 20:04:02.365655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.365622 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:02.365720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.365689 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:02.365784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.365718 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:02.971850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.971816 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" event={"ID":"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51","Type":"ContainerStarted","Data":"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d"} Nov 02 20:04:02.972310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:02.971874 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" event={"ID":"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51","Type":"ContainerStarted","Data":"6e5ddb6c2ed796ce3746f369a92fcfa72218a88d07257aedb38a376c35bc4bba"} Nov 02 20:04:03.090057 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:03.088085 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podStartSLOduration=2.088065839 podStartE2EDuration="2.088065839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:03.085810154 +0000 UTC m=+926.692354656" watchObservedRunningTime="2026-11-02 20:04:03.088065839 +0000 UTC m=+926.694610339" Nov 02 20:04:03.942081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:03.942042 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:03.943574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:03.943551 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:04:03.943686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:03.943596 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:04:04.942006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:04.941975 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:04:04.942393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:04.942030 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:04:05.941713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:05.941678 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:04:05.941912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:05.941748 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:04:06.941926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:06.941894 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:04:06.942349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:06.941946 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:04:07.941450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:07.941417 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:04:07.941624 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:07.941478 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:04:09.001355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:09.001308 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:09.002399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:09.002376 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:09.003300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:09.003271 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:04:21.005157 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:04:21.005115 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:04:33.434110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.434075 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 20:04:33.440767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.440721 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.490219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.490191 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c25349-openshift-default-dockercfg-5wzgl\"" Nov 02 20:04:33.539693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.539663 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 20:04:33.554847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.554811 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.554991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.554857 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.554991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.554954 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555058 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555126 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555099 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555125 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555181 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blt2d\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555223 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.555304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.555288 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656491 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656529 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656553 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656571 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blt2d\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656591 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656638 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656702 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656724 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.656827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.656793 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.657213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.657106 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.657271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.657218 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.657271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.657251 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.657378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.657291 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.657477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.657456 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.659009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.658983 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.659315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.659293 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.711176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.711148 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.711338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.711250 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blt2d\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:33.755132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:33.755102 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:34.171841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:34.171814 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 20:04:34.173425 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:04:34.173393 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8adf00_4cf7_4663_ae94_8f70534eb244.slice/crio-07cc85b41c76932cf13f7585a3f790d29239b27065b2e93e7a1a0742743afdc9 WatchSource:0}: Error finding container 07cc85b41c76932cf13f7585a3f790d29239b27065b2e93e7a1a0742743afdc9: Status 404 returned error can't find the container with id 07cc85b41c76932cf13f7585a3f790d29239b27065b2e93e7a1a0742743afdc9 Nov 02 20:04:34.175665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:34.175635 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:34.175756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:34.175698 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:34.175756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:34.175742 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:35.180423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.180390 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" event={"ID":"da8adf00-4cf7-4663-ae94-8f70534eb244","Type":"ContainerStarted","Data":"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f"} Nov 02 20:04:35.180849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.180438 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" event={"ID":"da8adf00-4cf7-4663-ae94-8f70534eb244","Type":"ContainerStarted","Data":"07cc85b41c76932cf13f7585a3f790d29239b27065b2e93e7a1a0742743afdc9"} Nov 02 20:04:35.400755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.388362 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podStartSLOduration=2.388343217 podStartE2EDuration="2.388343217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:35.338992418 +0000 UTC m=+958.945536917" watchObservedRunningTime="2026-11-02 20:04:35.388343217 +0000 UTC m=+958.994887716" Nov 02 20:04:35.755230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.755192 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:35.756627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.756606 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:04:35.756706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:35.756655 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:04:36.756354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:36.756322 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:04:36.756707 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:36.756370 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:04:37.757764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:37.756078 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:04:37.758294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:37.757927 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:04:38.756306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:38.756264 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:04:38.756525 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:38.756334 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:04:39.755934 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:39.755896 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:04:39.756452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:39.755977 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:04:40.972014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:40.971973 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:40.972527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:40.972505 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:40.973416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:40.973397 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 20:04:51.011881 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:04:51.011800 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:04:54.870988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:54.870959 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:04:54.874714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:54.874694 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:54.918320 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:54.918288 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:04:54.922164 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:54.922146 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:54.966535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:54.966511 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:04:55.013919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.013879 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.013973 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jg4\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014035 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014061 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014153 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014313 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014342 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014372 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gr2d\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014479 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014513 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014531 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014651 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014706 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.014818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014776 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.014818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014798 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.015047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014825 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.015047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014862 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.015047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.014890 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.024758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.024720 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:04:55.116193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116155 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116194 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jg4\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116327 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116364 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116400 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116460 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116488 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116517 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gr2d\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116570 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116611 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116638 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116714 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116773 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116812 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116845 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.116912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116876 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116911 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116925 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.116956 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117022 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117187 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117229 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.117502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117232 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117293 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117292 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117440 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117559 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.117836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.117800 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.119094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.119072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.119094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.119090 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.119410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.119391 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.119686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.119664 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.170343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.170263 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jg4\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.170343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.170279 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-s66gc\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.170533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.170516 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.171093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.171072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gr2d\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-n8w4g\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.188018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.187991 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:55.235182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.235152 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:55.603725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.603689 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:04:55.639798 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:04:55.639769 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f127d3_0a80_491d_acc1_48cf7e1cae8a.slice/crio-86e0a03ad1b9c17ec0eb78bd50f5a6285bd18e456e2b27ae6906649726768fdb WatchSource:0}: Error finding container 86e0a03ad1b9c17ec0eb78bd50f5a6285bd18e456e2b27ae6906649726768fdb: Status 404 returned error can't find the container with id 86e0a03ad1b9c17ec0eb78bd50f5a6285bd18e456e2b27ae6906649726768fdb Nov 02 20:04:55.641924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.641891 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:55.642006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.641951 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:55.642006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.641979 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:55.652303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.652281 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:04:55.653172 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:04:55.653137 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93ed1f6_54b8_4c85_8700_082702818464.slice/crio-7dbda460b5b5eea407dbf58b538fde8fe658e045f05ff44a9e859316a7103fe5 WatchSource:0}: Error finding container 7dbda460b5b5eea407dbf58b538fde8fe658e045f05ff44a9e859316a7103fe5: Status 404 returned error can't find the container with id 7dbda460b5b5eea407dbf58b538fde8fe658e045f05ff44a9e859316a7103fe5 Nov 02 20:04:55.655478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.655450 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:55.655560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.655508 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:55.655560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:55.655538 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:56.351714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.351673 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" event={"ID":"d93ed1f6-54b8-4c85-8700-082702818464","Type":"ContainerStarted","Data":"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1"} Nov 02 20:04:56.352107 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.351745 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" event={"ID":"d93ed1f6-54b8-4c85-8700-082702818464","Type":"ContainerStarted","Data":"7dbda460b5b5eea407dbf58b538fde8fe658e045f05ff44a9e859316a7103fe5"} Nov 02 20:04:56.353924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.353898 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" event={"ID":"c5f127d3-0a80-491d-acc1-48cf7e1cae8a","Type":"ContainerStarted","Data":"39d24f0f1f462e6a019e890eb029137a859adcc4ac0e489ba8e54c966c19bc66"} Nov 02 20:04:56.354028 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.353942 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" event={"ID":"c5f127d3-0a80-491d-acc1-48cf7e1cae8a","Type":"ContainerStarted","Data":"86e0a03ad1b9c17ec0eb78bd50f5a6285bd18e456e2b27ae6906649726768fdb"} Nov 02 20:04:56.469020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.468965 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podStartSLOduration=2.468949338 podStartE2EDuration="2.468949338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:56.4671286 +0000 UTC m=+980.073673092" watchObservedRunningTime="2026-11-02 20:04:56.468949338 +0000 UTC m=+980.075493836" Nov 02 20:04:56.582392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:56.582339 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podStartSLOduration=2.582327406 podStartE2EDuration="2.582327406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:56.581375348 +0000 UTC m=+980.187919847" watchObservedRunningTime="2026-11-02 20:04:56.582327406 +0000 UTC m=+980.188871904" Nov 02 20:04:57.197433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.192084 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:04:57.198834 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.198802 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:04:57.199049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.199021 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:04:57.237471 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.237435 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:04:57.237846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.237821 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:04:57.238051 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:57.238024 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:04:58.201754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:58.192893 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:04:58.201754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:58.192969 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:04:58.235757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:58.235550 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:04:58.235757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:58.235611 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:04:59.194763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:59.192099 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:04:59.194763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:59.192171 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:04:59.236085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:59.235953 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:04:59.236085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:04:59.236024 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:05:00.192171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:00.192124 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:05:00.192483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:00.192455 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:05:00.237142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:00.237105 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:05:00.237754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:00.237701 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:05:01.203755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:01.203372 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:05:01.203755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:01.203439 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:05:01.236260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:01.236226 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:05:01.236569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:01.236525 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:05:02.188576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:02.188528 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:05:02.189121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:02.188596 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:05:02.236050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:02.236010 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:05:02.236267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:02.236078 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:05:03.188828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:03.188790 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:05:03.189358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:03.189062 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:05:03.236503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:03.236470 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:05:03.236663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:03.236533 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:05:04.425705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.425674 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:05:04.426548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.426480 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:05:04.427331 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.427311 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:05:04.478182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.478152 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:05:04.478477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.478449 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:05:04.479313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:04.479296 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:05:09.994430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:09.994397 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:05:10.003543 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.003514 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.102748 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.102708 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:05:10.142634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142598 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142656 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142762 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfr5\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142783 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142802 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142824 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.142996 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142956 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.143030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.142997 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.143063 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.143051 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.243998 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfr5\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244034 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244057 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244078 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244126 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244163 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244220 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244279 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244344 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244530 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244640 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244681 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.244856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.244832 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.245022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.245004 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.246656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.246625 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.246791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.246776 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.298062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.298030 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.298206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.298188 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfr5\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dptmz\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.317078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.317052 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:10.738576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.738546 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:05:10.739446 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:05:10.739420 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bc9082_4e2a_4ca0_ba36_76bfe456605b.slice/crio-25b0c3955bc15331977489adac7569137780e07e81c8c2f3db5581f0da19e55b WatchSource:0}: Error finding container 25b0c3955bc15331977489adac7569137780e07e81c8c2f3db5581f0da19e55b: Status 404 returned error can't find the container with id 25b0c3955bc15331977489adac7569137780e07e81c8c2f3db5581f0da19e55b Nov 02 20:05:10.742207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.742174 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:10.742290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.742238 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:10.742290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:10.742267 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:11.441023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:11.440987 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" event={"ID":"37bc9082-4e2a-4ca0-ba36-76bfe456605b","Type":"ContainerStarted","Data":"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c"} Nov 02 20:05:11.441515 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:11.441049 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" event={"ID":"37bc9082-4e2a-4ca0-ba36-76bfe456605b","Type":"ContainerStarted","Data":"25b0c3955bc15331977489adac7569137780e07e81c8c2f3db5581f0da19e55b"} Nov 02 20:05:11.560868 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:11.560814 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podStartSLOduration=2.560799629 podStartE2EDuration="2.560799629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:11.558590807 +0000 UTC m=+995.165135306" watchObservedRunningTime="2026-11-02 20:05:11.560799629 +0000 UTC m=+995.167344126" Nov 02 20:05:12.318058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:12.318029 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:12.319535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:12.319510 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:12.319660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:12.319556 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:13.317802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:13.317765 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:13.318207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:13.317834 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:14.317493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:14.317452 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:14.317678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:14.317510 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:15.318377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:15.318337 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:15.318809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:15.318395 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:16.318191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:16.318150 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:16.318440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:16.318221 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:17.318013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:17.317980 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:17.318194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:17.318029 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:18.317835 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:18.317783 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:05:18.318255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:18.317857 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:05:19.321910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:19.321880 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:19.322296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:19.322180 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:19.322988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:05:19.322971 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:05:21.016128 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:05:21.016088 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:05:51.004121 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:05:51.004086 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:06:05.211200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.211166 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 20:06:05.215183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.215163 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.267725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.267695 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 20:06:05.285940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.285902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.286103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.285944 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.286103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.285970 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.286103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.286052 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.286283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.286116 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.286340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.286281 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2gb\" (UniqueName: \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.312965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.312938 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"openshift-service-ca.crt\"" Nov 02 20:06:05.312965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.312947 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"default-dockercfg-xfwx7\"" Nov 02 20:06:05.312965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.312963 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"kube-root-ca.crt\"" Nov 02 20:06:05.313147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.313030 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"distinct-w0-14d090-kserve-self-signed-certs\"" Nov 02 20:06:05.394509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.394463 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2gb\" (UniqueName: \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.396786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.396668 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397104 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397176 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397215 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397239 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397307 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.397990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.397932 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.398333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.398189 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.418109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.411175 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.418109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.412598 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.449052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.449026 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2gb\" (UniqueName: \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\") pod \"distinct-w0-14d090-kserve-d6bb57c57-5x7tl\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.527035 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.526967 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:05.930465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:05.930439 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 20:06:05.931416 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:06:05.931391 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0354fd75_2ef1_4e92_9751_dd1f945b4472.slice/crio-d79a9bebaa1a1d1947ad93603cd2ea3abd2ceddf13c08b9a142261d52b3d934b WatchSource:0}: Error finding container d79a9bebaa1a1d1947ad93603cd2ea3abd2ceddf13c08b9a142261d52b3d934b: Status 404 returned error can't find the container with id d79a9bebaa1a1d1947ad93603cd2ea3abd2ceddf13c08b9a142261d52b3d934b Nov 02 20:06:06.777226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:06.777180 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" event={"ID":"0354fd75-2ef1-4e92-9751-dd1f945b4472","Type":"ContainerStarted","Data":"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b"} Nov 02 20:06:06.778591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:06.778561 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" event={"ID":"0354fd75-2ef1-4e92-9751-dd1f945b4472","Type":"ContainerStarted","Data":"d79a9bebaa1a1d1947ad93603cd2ea3abd2ceddf13c08b9a142261d52b3d934b"} Nov 02 20:06:06.889205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:06.889159 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podStartSLOduration=1.889146113 podStartE2EDuration="1.889146113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:06.888300279 +0000 UTC m=+1050.494844796" watchObservedRunningTime="2026-11-02 20:06:06.889146113 +0000 UTC m=+1050.495690611" Nov 02 20:06:07.527079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:07.527044 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:15.530470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:15.529266 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:15.593753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:15.587869 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:15.903994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:15.903910 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:06:19.215497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.215462 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:06:19.219573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.219555 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289355 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhfc\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289403 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289500 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289603 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289632 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289664 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289761 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289803 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.289833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.289831 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.319094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.319067 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:06:19.391453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391422 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391455 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391488 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391525 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391606 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhfc\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391654 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391709 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391816 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391851 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.391971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.391959 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.392189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.392036 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.392242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.392184 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.392242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.392212 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.392464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.392441 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.393974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.393939 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.394194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.394175 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.445178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.445150 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.445322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.445264 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhfc\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.533262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.533180 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:19.963631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.963595 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:06:19.970806 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:06:19.970777 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde30c9f_9364_470c_8321_a0508ec544bc.slice/crio-3c494a0eeb820e910f3a04ed0779cb07d923625d1cb2cb4ffdf298cd2aa4e8d9 WatchSource:0}: Error finding container 3c494a0eeb820e910f3a04ed0779cb07d923625d1cb2cb4ffdf298cd2aa4e8d9: Status 404 returned error can't find the container with id 3c494a0eeb820e910f3a04ed0779cb07d923625d1cb2cb4ffdf298cd2aa4e8d9 Nov 02 20:06:19.973131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.973101 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:19.973228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.973174 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:19.973228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:19.973202 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:20.925089 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:20.925043 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" event={"ID":"cde30c9f-9364-470c-8321-a0508ec544bc","Type":"ContainerStarted","Data":"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b"} Nov 02 20:06:20.925566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:20.925109 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" event={"ID":"cde30c9f-9364-470c-8321-a0508ec544bc","Type":"ContainerStarted","Data":"3c494a0eeb820e910f3a04ed0779cb07d923625d1cb2cb4ffdf298cd2aa4e8d9"} Nov 02 20:06:21.024707 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:06:21.024672 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:06:21.533907 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:21.533873 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:21.535457 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:21.535429 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:21.535590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:21.535484 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:22.534109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:22.534071 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:22.534524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:22.534123 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:23.534436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:23.534359 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:23.534436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:23.534410 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:24.534094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:24.534056 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:24.534243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:24.534109 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:25.534185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:25.534149 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:25.534702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:25.534207 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:26.533912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:26.533873 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:06:26.534099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:26.533930 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:06:27.537961 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:27.537930 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:27.538327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:27.538263 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:27.538966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:27.538949 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:06:27.696090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:27.696035 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podStartSLOduration=8.696021852 podStartE2EDuration="8.696021852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:21.129836023 +0000 UTC m=+1064.736380512" watchObservedRunningTime="2026-11-02 20:06:27.696021852 +0000 UTC m=+1071.302566350" Nov 02 20:06:34.453592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.453563 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:06:34.457477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.457460 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.464713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.464687 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.464749 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.464932 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.464985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.465058 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfll\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.465105 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.465136 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.465152 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.465386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.465169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.507212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.507186 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\"" Nov 02 20:06:34.557136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.557106 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:06:34.565856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.565824 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.565874 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.565911 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566013 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566072 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566113 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566226 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566266 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566288 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566346 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfll\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566497 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566261 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.566833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.566812 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.567223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.567200 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.568962 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.568940 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.569092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.569072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.619637 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.619608 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.619795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.619761 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfll\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:34.770889 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:34.770810 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:35.187019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:35.186983 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:06:35.187406 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:06:35.187376 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f7d876_5c3f_4ea0_9fae_cd555d11a60c.slice/crio-e98f08bdb719289dc9a704cbf416764dbb3b4f97715621fea08805ac18346703 WatchSource:0}: Error finding container e98f08bdb719289dc9a704cbf416764dbb3b4f97715621fea08805ac18346703: Status 404 returned error can't find the container with id e98f08bdb719289dc9a704cbf416764dbb3b4f97715621fea08805ac18346703 Nov 02 20:06:35.189937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:35.189903 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:35.190008 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:35.189976 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:35.190045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:35.190010 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:36.070038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.069997 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" event={"ID":"89f7d876-5c3f-4ea0-9fae-cd555d11a60c","Type":"ContainerStarted","Data":"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4"} Nov 02 20:06:36.070456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.070046 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" event={"ID":"89f7d876-5c3f-4ea0-9fae-cd555d11a60c","Type":"ContainerStarted","Data":"e98f08bdb719289dc9a704cbf416764dbb3b4f97715621fea08805ac18346703"} Nov 02 20:06:36.228434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.228379 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podStartSLOduration=2.228365494 podStartE2EDuration="2.228365494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:36.226673249 +0000 UTC m=+1079.833217748" watchObservedRunningTime="2026-11-02 20:06:36.228365494 +0000 UTC m=+1079.834910066" Nov 02 20:06:36.771873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.771837 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:36.773224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.773195 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:36.773345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:36.773243 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:37.771798 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:37.771767 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:37.772188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:37.771817 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:38.771097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:38.771060 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:38.771269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:38.771110 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:39.771824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:39.771791 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:39.772246 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:39.771845 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:40.771698 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:40.771653 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:40.771923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:40.771750 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:41.771661 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:41.771625 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:06:41.771866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:41.771691 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:06:42.775933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:42.775905 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:42.776328 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:42.776170 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:42.776825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:06:42.776806 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:06:51.002675 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:06:51.002637 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:07:21.003647 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:07:21.003608 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:07:51.003965 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:07:51.003923 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:07:58.336696 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.336662 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 20:07:58.341175 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.341154 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.393533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.393497 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 20:07:58.395656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.395626 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.395849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.395666 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.395849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.395782 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.395849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.395817 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.396039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.395969 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ps82\" (UniqueName: \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.396039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.396013 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.437385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.437363 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"openshift-service-ca.crt\"" Nov 02 20:07:58.437385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.437366 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"kube-root-ca.crt\"" Nov 02 20:07:58.437547 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.437369 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"default-dockercfg-hxc6x\"" Nov 02 20:07:58.438498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.438484 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"distinct-2-w2-426941-kserve-self-signed-certs\"" Nov 02 20:07:58.504250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.504208 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ps82\" (UniqueName: \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.530274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.530244 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.531339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.531313 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.531954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.531936 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.532436 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.541859 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.532340 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.531884 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.531054 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.551766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.544804 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.552898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.552860 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.585894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.585864 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ps82\" (UniqueName: \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\") pod \"distinct-2-w2-426941-kserve-68b8c94948-bsr58\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:58.653045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:58.652966 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:07:59.017722 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:59.017698 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 20:07:59.659951 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:59.659915 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" event={"ID":"b4053b5f-e54a-405a-bbf9-bd67a1650d4a","Type":"ContainerStarted","Data":"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c"} Nov 02 20:07:59.660363 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:59.659964 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" event={"ID":"b4053b5f-e54a-405a-bbf9-bd67a1650d4a","Type":"ContainerStarted","Data":"38f47d451a1f25f92942b07d776f885c9f7950c18a29c8c268ce5f1862ea9261"} Nov 02 20:07:59.769577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:07:59.769520 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podStartSLOduration=1.769501912 podStartE2EDuration="1.769501912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:59.767263329 +0000 UTC m=+1163.373807828" watchObservedRunningTime="2026-11-02 20:07:59.769501912 +0000 UTC m=+1163.376046411" Nov 02 20:08:00.654935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:00.654871 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:08:08.654077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:08.654043 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:08:08.666292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:08.666267 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:08:08.738845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:08.738802 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:08:21.004201 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:08:21.004106 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:08:33.643520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.643487 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:08:33.648197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.648172 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.738285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.738251 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:08:33.738897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.738871 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739055 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739116 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqf4\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739229 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739268 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739361 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739408 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739500 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.739590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.739558 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.840965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.840935 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqf4\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.840987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841004 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841104 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841151 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841185 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841215 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841268 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841383 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841386 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841449 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841633 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.841715 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.841697 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.842043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.842024 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.843449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.843432 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.843579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.843560 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.896054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.895985 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqf4\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.896054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.895992 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:33.966193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:33.966154 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:34.383237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.383191 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:08:34.419242 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:08:34.419214 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5feec8_d535_4a7e_a1ea_33999b4f9978.slice/crio-aef9e2ad522eb471faa8423b0e7f98ebbd79cc2b9fdaa6a593ef5409c0c04735 WatchSource:0}: Error finding container aef9e2ad522eb471faa8423b0e7f98ebbd79cc2b9fdaa6a593ef5409c0c04735: Status 404 returned error can't find the container with id aef9e2ad522eb471faa8423b0e7f98ebbd79cc2b9fdaa6a593ef5409c0c04735 Nov 02 20:08:34.422318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.422274 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:34.422409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.422380 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:34.422457 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.422421 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:34.907877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.907844 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" event={"ID":"eb5feec8-d535-4a7e-a1ea-33999b4f9978","Type":"ContainerStarted","Data":"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6"} Nov 02 20:08:34.908277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.907904 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" event={"ID":"eb5feec8-d535-4a7e-a1ea-33999b4f9978","Type":"ContainerStarted","Data":"aef9e2ad522eb471faa8423b0e7f98ebbd79cc2b9fdaa6a593ef5409c0c04735"} Nov 02 20:08:34.966336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:34.966310 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:35.020635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.020583 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podStartSLOduration=2.020568953 podStartE2EDuration="2.020568953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:35.019132596 +0000 UTC m=+1198.625677095" watchObservedRunningTime="2026-11-02 20:08:35.020568953 +0000 UTC m=+1198.627113506" Nov 02 20:08:35.185574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.185494 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:08:35.216119 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.216086 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258063 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258029 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258146 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258206 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258383 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258425 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4bjw\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258482 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258585 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.258664 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.258645 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.380772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.371897 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.387356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.387319 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.387587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.387560 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.387784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.387765 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.397882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.397840 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.398215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.398189 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bjw\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.398378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.398360 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.398535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.398521 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.398640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.398627 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.400897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.400869 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.401049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.380192 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.401485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.401399 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.401611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.381357 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:08:35.402385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.402365 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.403930 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.403149 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.410053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.410028 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.419341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.419308 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.457885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.457856 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.458065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.458045 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4bjw\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.533436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.533406 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:35.969035 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.969004 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:35.969755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.969056 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:35.972497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.972470 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:08:35.973187 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:08:35.973162 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027a52bb_63d6_49fc_bf68_eb2727f275b9.slice/crio-9d1036c06ef055e0b6b760c3bf9c398cdd19c56cbdcacc38908f71f9b2371c08 WatchSource:0}: Error finding container 9d1036c06ef055e0b6b760c3bf9c398cdd19c56cbdcacc38908f71f9b2371c08: Status 404 returned error can't find the container with id 9d1036c06ef055e0b6b760c3bf9c398cdd19c56cbdcacc38908f71f9b2371c08 Nov 02 20:08:35.975604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.975576 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:35.975670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.975636 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:35.975670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:35.975666 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:36.946143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:36.945919 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" event={"ID":"027a52bb-63d6-49fc-bf68-eb2727f275b9","Type":"ContainerStarted","Data":"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82"} Nov 02 20:08:36.946143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:36.945986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" event={"ID":"027a52bb-63d6-49fc-bf68-eb2727f275b9","Type":"ContainerStarted","Data":"9d1036c06ef055e0b6b760c3bf9c398cdd19c56cbdcacc38908f71f9b2371c08"} Nov 02 20:08:36.966963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:36.966936 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:36.967104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:36.966981 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:37.163411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.163359 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podStartSLOduration=2.163345256 podStartE2EDuration="2.163345256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:37.130168278 +0000 UTC m=+1200.736712790" watchObservedRunningTime="2026-11-02 20:08:37.163345256 +0000 UTC m=+1200.769889753" Nov 02 20:08:37.537805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.537483 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:37.537805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.537653 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:37.537805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.537698 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:37.967299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.967269 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:37.967516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:37.967322 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:38.537756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:38.537168 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:38.537756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:38.537238 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:38.966438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:38.966410 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:38.966744 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:38.966669 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:39.536757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:39.535157 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:39.536757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:39.535221 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:39.967844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:39.967809 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:39.968499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:39.967855 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:40.534030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:40.533996 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:40.534255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:40.534058 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:40.967374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:40.967341 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:08:40.967563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:40.967402 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 20:08:41.534208 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:41.534167 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:41.534622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:41.534232 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:41.974960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:41.974926 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:41.975365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:41.975342 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:41.976081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:41.976063 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:08:42.534535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:42.534497 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:08:42.534943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:42.534561 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 20:08:43.538135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:43.538105 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:43.538509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:43.538478 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:43.539363 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:08:43.539342 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:08:51.004356 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:08:51.004317 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:09:13.600976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:13.600933 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:13.601776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:13.601740 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.878918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:13.878827 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:13.891754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:13.879155 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:19.563318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.563284 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 20:09:19.568797 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.568770 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.620564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.620535 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 20:09:19.627887 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.627853 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.628034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.627986 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.628089 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.628043 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b577m\" (UniqueName: \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.628191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.628174 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.628255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.628208 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.628386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.628360 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.643807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.643788 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"openshift-service-ca.crt\"" Nov 02 20:09:19.663706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.663681 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"kube-root-ca.crt\"" Nov 02 20:09:19.665691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.665668 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"distinct-2-w1-bbee67-kserve-self-signed-certs\"" Nov 02 20:09:19.665822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.665699 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"default-dockercfg-gj7fz\"" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.757984 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.758095 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.758133 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b577m\" (UniqueName: \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.758206 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.758236 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.764770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.758317 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.765670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.765605 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.787877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.771259 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.787877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.773227 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.787877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.784446 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.789231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.789126 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.827533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.827456 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b577m\" (UniqueName: \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\") pod \"distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:19.881449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:19.881420 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:20.287281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.287255 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 20:09:20.288320 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:09:20.288295 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d04e1e6_40a4_4ee4_80d0_c6ff72f91f61.slice/crio-84723b8f55eb4b0a97858adb4321bdc9265c5594472c4cd55ba2bfd6b254470a WatchSource:0}: Error finding container 84723b8f55eb4b0a97858adb4321bdc9265c5594472c4cd55ba2bfd6b254470a: Status 404 returned error can't find the container with id 84723b8f55eb4b0a97858adb4321bdc9265c5594472c4cd55ba2bfd6b254470a Nov 02 20:09:20.344476 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.344445 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" event={"ID":"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61","Type":"ContainerStarted","Data":"84723b8f55eb4b0a97858adb4321bdc9265c5594472c4cd55ba2bfd6b254470a"} Nov 02 20:09:20.643289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.643215 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 20:09:20.647383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.647362 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.702198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.702169 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 20:09:20.782946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.782759 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lj6m\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.782946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.782879 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.782946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.782918 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783066 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783121 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783236 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783273 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783365 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.783440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.783420 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.884981 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885042 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885090 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885141 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885166 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885207 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885296 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885372 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lj6m\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885480 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885556 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.885760 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885670 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.886009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.885969 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.886152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.886019 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.886556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.886532 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.888034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.888009 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.888229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.888208 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.939256 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.939187 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.939567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.939547 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lj6m\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:20.961468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:20.961446 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:21.003149 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:09:21.003108 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:09:21.352678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.352642 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" event={"ID":"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61","Type":"ContainerStarted","Data":"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955"} Nov 02 20:09:21.381712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.381690 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 20:09:21.382204 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:09:21.382180 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7bdf44_35b1_4525_91d9_3e35fb6ec5b5.slice/crio-b2aa8cfcbf5fc6b3957d9dffc79e5d6739567c05f6530477425e98c1de441361 WatchSource:0}: Error finding container b2aa8cfcbf5fc6b3957d9dffc79e5d6739567c05f6530477425e98c1de441361: Status 404 returned error can't find the container with id b2aa8cfcbf5fc6b3957d9dffc79e5d6739567c05f6530477425e98c1de441361 Nov 02 20:09:21.384670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.384639 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:21.384762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.384704 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:21.384813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.384769 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:21.462360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.462309 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podStartSLOduration=2.462296918 podStartE2EDuration="2.462296918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:21.460151115 +0000 UTC m=+1245.066695600" watchObservedRunningTime="2026-11-02 20:09:21.462296918 +0000 UTC m=+1245.068841416" Nov 02 20:09:21.882114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:21.882081 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:22.359554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.359517 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" event={"ID":"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5","Type":"ContainerStarted","Data":"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf"} Nov 02 20:09:22.359762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.359568 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" event={"ID":"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5","Type":"ContainerStarted","Data":"b2aa8cfcbf5fc6b3957d9dffc79e5d6739567c05f6530477425e98c1de441361"} Nov 02 20:09:22.568884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.568836 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podStartSLOduration=2.568820223 podStartE2EDuration="2.568820223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:22.521320394 +0000 UTC m=+1246.127864904" watchObservedRunningTime="2026-11-02 20:09:22.568820223 +0000 UTC m=+1246.175364723" Nov 02 20:09:22.961884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.961849 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:22.963795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.963767 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:22.963915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:22.963813 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:23.962459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:23.962423 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:23.962856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:23.962473 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:24.962511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:24.962477 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:24.962957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:24.962541 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:25.962615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:25.962579 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:25.963102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:25.962650 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:26.962438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:26.962410 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:26.962614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:26.962458 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:27.962171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:27.962126 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:09:27.962677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:27.962187 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:09:28.966369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:28.966341 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:28.966868 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:28.966600 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:28.967395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:28.967380 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 20:09:29.881854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:29.881816 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:29.901424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:29.901396 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:30.419691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:30.419660 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:34.086459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.086429 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:09:34.090785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.090765 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.134421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.134397 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:09:34.138214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.138199 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.163248 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163225 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163265 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163291 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4hj\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163425 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163468 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163528 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163621 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163609 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163633 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.163692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.163657 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.218722 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.218686 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:09:34.264784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.264709 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.264963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.264824 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxf8\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.264963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.264898 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.264963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.264950 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265029 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265076 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265104 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.265288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265139 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.265288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265187 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.265288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265218 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265250 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265443 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265292 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265791 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265876 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.265953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.265944 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.266095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266013 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.266153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266114 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.266207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266160 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.266207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266190 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.266319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266227 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4hj\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.266376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266345 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.266699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266673 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.266818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.266722 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.267991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.267969 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.268220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.268200 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.280825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.280796 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:09:34.322337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.322304 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.322955 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.322913 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4hj\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.368216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368119 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368208 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368279 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368316 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368422 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368487 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368525 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxf8\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368561 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368651 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368675 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.368922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368893 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.369001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.368922 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.369141 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.369105 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.369432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.369410 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.370964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.370947 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.371199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.371182 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.404595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.404571 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:34.422868 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.422823 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.423416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.423387 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxf8\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.451897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.451869 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:34.823044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.823020 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:09:34.824300 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:09:34.824274 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6194de7e_caca_4b06_a83a_6a6a492b6bca.slice/crio-303481d4de85d7f2b416f34674dc0a8b557d1a4ebf1688e79359369f13030a63 WatchSource:0}: Error finding container 303481d4de85d7f2b416f34674dc0a8b557d1a4ebf1688e79359369f13030a63: Status 404 returned error can't find the container with id 303481d4de85d7f2b416f34674dc0a8b557d1a4ebf1688e79359369f13030a63 Nov 02 20:09:34.826804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.826764 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:34.826893 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.826852 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:34.826939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.826898 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:34.870753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.870717 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:09:34.887090 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:09:34.887050 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f166cc2_1183_4da4_94c3_6fcb437fbade.slice/crio-d5a0fe1a919a262cb600d12d8ad5ee3961b831c39d78bc2fd47f2c86a88d494a WatchSource:0}: Error finding container d5a0fe1a919a262cb600d12d8ad5ee3961b831c39d78bc2fd47f2c86a88d494a: Status 404 returned error can't find the container with id d5a0fe1a919a262cb600d12d8ad5ee3961b831c39d78bc2fd47f2c86a88d494a Nov 02 20:09:34.889969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.889936 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:34.890062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.890016 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:34.890131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:34.890063 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:35.440644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.440609 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" event={"ID":"1f166cc2-1183-4da4-94c3-6fcb437fbade","Type":"ContainerStarted","Data":"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17"} Nov 02 20:09:35.441171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.440670 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" event={"ID":"1f166cc2-1183-4da4-94c3-6fcb437fbade","Type":"ContainerStarted","Data":"d5a0fe1a919a262cb600d12d8ad5ee3961b831c39d78bc2fd47f2c86a88d494a"} Nov 02 20:09:35.442357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.442331 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" event={"ID":"6194de7e-caca-4b06-a83a-6a6a492b6bca","Type":"ContainerStarted","Data":"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726"} Nov 02 20:09:35.442463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.442378 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" event={"ID":"6194de7e-caca-4b06-a83a-6a6a492b6bca","Type":"ContainerStarted","Data":"303481d4de85d7f2b416f34674dc0a8b557d1a4ebf1688e79359369f13030a63"} Nov 02 20:09:35.452779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.452757 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:35.454523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.454480 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:35.454604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.454523 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:35.562675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.562619 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podStartSLOduration=1.562605502 podStartE2EDuration="1.562605502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:35.560999467 +0000 UTC m=+1259.167543966" watchObservedRunningTime="2026-11-02 20:09:35.562605502 +0000 UTC m=+1259.169150000" Nov 02 20:09:35.678934 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:35.678718 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podStartSLOduration=1.678691921 podStartE2EDuration="1.678691921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:35.675379242 +0000 UTC m=+1259.281923738" watchObservedRunningTime="2026-11-02 20:09:35.678691921 +0000 UTC m=+1259.285236420" Nov 02 20:09:36.411755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:36.410828 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:36.419753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:36.418755 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:36.419753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:36.418907 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:36.452405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:36.452311 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:36.452405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:36.452396 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:37.406751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:37.406118 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:37.406751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:37.406198 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:37.458689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:37.458653 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:37.459321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:37.459291 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:38.414757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:38.406219 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:38.414757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:38.406297 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:38.456756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:38.453099 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:38.456756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:38.453162 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:39.411886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.411834 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.413709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.413683 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:39.413926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.413901 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:39.453141 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.453108 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:39.453389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.453358 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:39.695818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.682350 5136 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.794766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.788108 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:09:40.067773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.894947 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.067773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:39.894983 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 20:09:40.172786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.169902 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:40.172786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.169986 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.424857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.414152 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.424857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.415092 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:40.424857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.419049 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:40.424857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.419097 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:40.475755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.461939 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:40.475755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.462001 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:40.993937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:40.934819 5136 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.040971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.040896 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.041772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.041539 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.044266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.042196 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:09:41.044266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.042256 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:09:41.044266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.041751 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.044266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.044166 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:09:41.287183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.287101 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:09:41.425892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.425854 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.441209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.430579 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:41.442457 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.442422 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:41.475006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.474892 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:41.475006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.474949 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:41.564074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.563996 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:09:41.579150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.579128 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:09:41.585993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:41.585966 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:09:42.409102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:42.405100 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:42.409102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:42.405183 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:42.454750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:42.454504 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:42.454750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:42.454580 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:43.411159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:43.411120 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:09:43.411459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:43.411430 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:09:43.452435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:43.452402 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:43.452571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:43.452439 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:44.454592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:44.454146 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:09:44.454592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:44.454229 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 20:09:45.136069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:45.136037 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:45.136835 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:45.136715 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:45.137838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:45.137816 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:09:46.493899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:46.493819 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:46.493899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:46.493875 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:46.845007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:46.844923 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:46.845261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:46.845239 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:46.847003 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:46.846977 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:09:51.004378 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:09:51.004339 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:09:55.934762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:55.931907 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:09:55.953458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:55.953426 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.050309 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.049853 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.050914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.050797 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58d46\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.051374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.051222 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.052274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.052251 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.054346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.052435 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.054768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.054722 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.064862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.062154 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:09:56.065088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.065067 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.065266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.065249 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.065386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.065367 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167008 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.166977 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167008 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167014 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167047 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167086 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167101 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167247 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167326 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58d46\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167436 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167511 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167518 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167552 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167582 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167597 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.167839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.167812 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.169818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.169796 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.169960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.169941 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.221368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.221336 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.221522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.221447 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58d46\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.296501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.296467 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:56.717812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.717783 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:09:56.722900 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.722865 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:56.723032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.722941 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:56.723032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:56.722974 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:09:57.228173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.228129 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" event={"ID":"d54ea449-0c01-48ec-91b2-ad42b3830434","Type":"ContainerStarted","Data":"54c75d37c38d94a7a6ead835fe255fd0d26b4088e3ca85300a9743fa1beb286c"} Nov 02 20:09:57.228713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.228693 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" event={"ID":"d54ea449-0c01-48ec-91b2-ad42b3830434","Type":"ContainerStarted","Data":"38cb387138a8c2e67efc1ec70e3600a239805ebe0b0a5470634032946d6d1c02"} Nov 02 20:09:57.297345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.297316 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:09:57.299408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.299379 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:09:57.299564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.299433 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:09:57.343718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:57.343648 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podStartSLOduration=2.3436297919999998 podStartE2EDuration="2.343629792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:57.340695713 +0000 UTC m=+1280.947240226" watchObservedRunningTime="2026-11-02 20:09:57.343629792 +0000 UTC m=+1280.950174293" Nov 02 20:09:58.306756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:58.297075 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:09:58.306756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:58.297138 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:09:59.312889 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:59.312844 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:09:59.313415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:09:59.312907 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:00.308755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:00.305821 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:00.308755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:00.305881 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:01.307755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:01.298183 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:01.307755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:01.298271 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:02.309705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:02.302851 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:02.309705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:02.302916 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:03.301757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:03.301671 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:03.321756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:03.318280 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:04.310756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:04.309827 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:10:04.333782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:04.311115 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:04.333782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:04.311167 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:04.437759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:04.434339 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.582214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:04.582114 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 20:10:05.318758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.317862 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.318758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.318660 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:10:05.343362 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.343323 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:05.371982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.371913 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:05.394784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.385853 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:05.394784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.385917 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.441873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.441829 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.442485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.442459 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:10:05.609814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.595675 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:05.610101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.610063 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.611157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.611119 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.612391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.612366 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:10:05.656872 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:05.656820 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 20:10:06.163390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.163350 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:06.163724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.163677 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.318293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.318248 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.412659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.412619 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:10:06.413281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.413248 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:10:06.462832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.456359 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:10:06.619873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.619825 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.630140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.630099 5136 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.636802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.636772 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:10:06.647142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.647114 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:10:06.647454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.647434 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.657925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.657879 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.659172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.658306 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:10:06.752498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.744427 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836641 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836672 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836719 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836790 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836858 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836942 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.836992 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.837931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.837015 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrvs\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946527 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946611 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946661 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946684 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrvs\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946809 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946886 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946912 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.946957 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.947011 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.952692 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.952999 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.958920 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.964130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.959203 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:06.990706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:06.986427 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.131632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.122280 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.155274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.155235 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.244272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.244235 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrvs\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.327756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.323344 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.327756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.323441 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.562206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.562163 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.596799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.588308 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:07.669395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.668513 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 20:10:07.703185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.703152 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:07.703475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.703409 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.711039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.711012 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 20:10:07.775775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:07.760976 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:08.286861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:08.277115 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 20:10:08.381755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:08.378170 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:08.406752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:08.382023 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:08.863762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:08.847846 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:09.050337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.050291 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:10:09.713559 5136 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.139s" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.713637 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.713761 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.713902 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.713931 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.713971 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.714109 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:09.714214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.714201 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:09.714616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.714252 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:09.715201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.714890 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:10:09.721231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:09.721141 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:10:10.291455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.291414 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" event={"ID":"659b69c2-464c-4ed8-a7c2-8178fc81a723","Type":"ContainerStarted","Data":"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3"} Nov 02 20:10:10.291986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.291488 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" event={"ID":"659b69c2-464c-4ed8-a7c2-8178fc81a723","Type":"ContainerStarted","Data":"581aaf1c9de37f602a39758ff7cb385ea2dc09e10f753c38e87192ff18ce5fdc"} Nov 02 20:10:10.421817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.421712 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podStartSLOduration=5.421686701 podStartE2EDuration="5.421686701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:10.403248226 +0000 UTC m=+1294.009792804" watchObservedRunningTime="2026-11-02 20:10:10.421686701 +0000 UTC m=+1294.028231202" Nov 02 20:10:10.601811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.601623 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:10.616751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.605888 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:10.616751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:10.605967 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:11.606752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:11.606380 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:11.606752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:11.606453 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:12.610755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:12.604100 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:12.610755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:12.604169 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:13.609756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:13.609657 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:13.609756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:13.609717 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:14.606754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:14.604079 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:14.606754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:14.604162 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:15.602753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:15.598042 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:15.602753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:15.598112 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:16.598620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:16.598585 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:10:16.599020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:16.598642 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:10:17.602959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:17.602928 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:17.603354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:17.603294 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:17.604133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:17.604117 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:10:20.558228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.558200 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:10:20.562489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.562469 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.607890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.607861 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:10:20.645503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.645472 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpn4\" (UniqueName: \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.645682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.645652 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.645877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.645858 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.645951 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.645892 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.652316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.652295 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5e04421b-serving-cert\"" Nov 02 20:10:20.747417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.747379 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.747599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.747503 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.747599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.747532 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.747687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.747629 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpn4\" (UniqueName: \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.747687 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:10:20.747529 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-serving-cert: secret "maas-api-e2e-mt-5e04421b-serving-cert" not found Nov 02 20:10:20.747777 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:10:20.747741 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls podName:496d118d-3398-4f92-aa8c-d8cc19c77d3b nodeName:}" failed. No retries permitted until 2026-11-02 20:10:21.247701116 +0000 UTC m=+1304.854245609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls") pod "maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" (UID: "496d118d-3398-4f92-aa8c-d8cc19c77d3b") : secret "maas-api-e2e-mt-5e04421b-serving-cert" not found Nov 02 20:10:20.748206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.748186 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.750000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.749979 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:20.836816 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:20.836719 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpn4\" (UniqueName: \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:21.068980 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:10:21.068940 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:10:21.262564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:21.262529 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:21.269298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:21.269267 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-kxn5p\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:21.476407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:21.476376 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:21.846381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:21.846345 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:10:21.846844 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:10:21.846458 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d118d_3398_4f92_aa8c_d8cc19c77d3b.slice/crio-4b4e8a385f485bf5824c80a97f0d973d07be43c2df4251c1ea101a7a2137cc74 WatchSource:0}: Error finding container 4b4e8a385f485bf5824c80a97f0d973d07be43c2df4251c1ea101a7a2137cc74: Status 404 returned error can't find the container with id 4b4e8a385f485bf5824c80a97f0d973d07be43c2df4251c1ea101a7a2137cc74 Nov 02 20:10:21.848424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:21.848406 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:10:22.388751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:22.388699 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" event={"ID":"496d118d-3398-4f92-aa8c-d8cc19c77d3b","Type":"ContainerStarted","Data":"4b4e8a385f485bf5824c80a97f0d973d07be43c2df4251c1ea101a7a2137cc74"} Nov 02 20:10:23.397434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:23.397397 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" event={"ID":"496d118d-3398-4f92-aa8c-d8cc19c77d3b","Type":"ContainerStarted","Data":"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479"} Nov 02 20:10:23.397937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:23.397618 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:23.509872 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:23.509808 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" podStartSLOduration=2.582049023 podStartE2EDuration="3.509789658s" totalImagesPullingTime="927.740635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:23.508313117 +0000 UTC m=+1307.114857616" watchObservedRunningTime="2026-11-02 20:10:23.509789658 +0000 UTC m=+1307.116334178" Nov 02 20:10:29.409276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:29.409242 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:10:48.186496 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.186450 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:10:48.192038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.192016 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.307705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.307661 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.311753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.308201 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.319007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.318713 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.322007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.321965 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.322305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.322285 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt42g\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.322494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.322478 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.322804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.322777 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.322975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.322959 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.323167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.323150 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.338258 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.338168 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-08eeaa-openshift-default-dockercfg-8hhv5\"" Nov 02 20:10:48.360757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.360706 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:10:48.437573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.437493 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.438362 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.438336 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.438601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.438584 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.438754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.438720 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.438920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.438901 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt42g\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.439052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439034 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439136 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439266 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439295 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439356 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.439702 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.441359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.440003 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.442198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.442147 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.442578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.442528 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.448045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.448020 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.448989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.448966 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.493515 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.493488 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.494041 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.494018 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt42g\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\") pod \"e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:48.512589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:48.512568 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:49.022614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.022580 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:10:49.058624 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:10:49.058594 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2601ac1b_87cc_42ef_8089_f36a3111d77c.slice/crio-dde5ac0f17bbd47f005b7f501987241709aae671d19cc3bdf9e36df6887f4bf7 WatchSource:0}: Error finding container dde5ac0f17bbd47f005b7f501987241709aae671d19cc3bdf9e36df6887f4bf7: Status 404 returned error can't find the container with id dde5ac0f17bbd47f005b7f501987241709aae671d19cc3bdf9e36df6887f4bf7 Nov 02 20:10:49.061104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.061067 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:49.061224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.061155 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:49.061224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.061200 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:49.690235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.690198 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerStarted","Data":"eb6ca35d90f557548174c988dbcca9da5326465deba04d0f9a04b9fe8cf22f80"} Nov 02 20:10:49.690635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.690257 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerStarted","Data":"dde5ac0f17bbd47f005b7f501987241709aae671d19cc3bdf9e36df6887f4bf7"} Nov 02 20:10:49.813919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:49.813839 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podStartSLOduration=1.8138249850000001 podStartE2EDuration="1.813824985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:49.811278892 +0000 UTC m=+1333.417823395" watchObservedRunningTime="2026-11-02 20:10:49.813824985 +0000 UTC m=+1333.420369482" Nov 02 20:10:50.513640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:50.513605 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:50.519212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:50.519182 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:50.696396 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:50.696368 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:50.697257 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:10:50.697239 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:10:51.007891 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:10:51.007856 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:11:17.650454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.650374 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:11:17.654890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.654873 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.706788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.706758 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:11:17.750330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.750304 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-45c182-serving-cert\"" Nov 02 20:11:17.781633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.781607 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.781808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.781668 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.781850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.781806 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.781885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.781867 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2tl\" (UniqueName: \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.882832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.882799 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.883015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.882900 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2tl\" (UniqueName: \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.883015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.882991 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.883144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.883057 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.883244 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:17.883223 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-serving-cert: secret "maas-api-e2e-api-a-45c182-serving-cert" not found Nov 02 20:11:17.883300 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:17.883289 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls podName:822a4355-c346-4689-bfe2-21344dedf120 nodeName:}" failed. No retries permitted until 2026-11-02 20:11:18.383268093 +0000 UTC m=+1361.989812578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls") pod "maas-api-e2e-api-a-45c182-5765588bbd-97ltv" (UID: "822a4355-c346-4689-bfe2-21344dedf120") : secret "maas-api-e2e-api-a-45c182-serving-cert" not found Nov 02 20:11:17.883442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.883424 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.885457 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.885430 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:17.978755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:17.972713 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2tl\" (UniqueName: \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:18.427486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:18.427395 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:18.429914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:18.429891 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") pod \"maas-api-e2e-api-a-45c182-5765588bbd-97ltv\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:18.569076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:18.569046 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:18.937172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:18.937109 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:11:18.937883 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:11:18.937377 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822a4355_c346_4689_bfe2_21344dedf120.slice/crio-eec046031cc755dd9b41eba17dc4dbc636e051e3b041542ca5c2646fd3da1bf0 WatchSource:0}: Error finding container eec046031cc755dd9b41eba17dc4dbc636e051e3b041542ca5c2646fd3da1bf0: Status 404 returned error can't find the container with id eec046031cc755dd9b41eba17dc4dbc636e051e3b041542ca5c2646fd3da1bf0 Nov 02 20:11:19.027886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:19.027852 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" event={"ID":"822a4355-c346-4689-bfe2-21344dedf120","Type":"ContainerStarted","Data":"eec046031cc755dd9b41eba17dc4dbc636e051e3b041542ca5c2646fd3da1bf0"} Nov 02 20:11:20.035870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:20.035796 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" event={"ID":"822a4355-c346-4689-bfe2-21344dedf120","Type":"ContainerStarted","Data":"c68cc10a47cb84ab21764fd2fef2cf5624757bc9e4529abdfed2b0112d46ab2b"} Nov 02 20:11:20.036237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:20.035930 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:20.148082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:20.148019 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" podStartSLOduration=2.328334099 podStartE2EDuration="3.148001875s" totalImagesPullingTime="819.667776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:20.146882347 +0000 UTC m=+1363.753426845" watchObservedRunningTime="2026-11-02 20:11:20.148001875 +0000 UTC m=+1363.754546374" Nov 02 20:11:21.004059 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:21.004020 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:11:26.047803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:26.047777 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:11:32.555503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.555474 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:11:32.559567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.559547 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.612968 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.612937 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:11:32.656354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.656318 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-08eeaa-serving-cert\"" Nov 02 20:11:32.675027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.674998 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.675164 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.675058 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58m48\" (UniqueName: \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.675212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.675172 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.675336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.675318 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.776315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.776282 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.776476 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.776393 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.776476 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.776454 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58m48\" (UniqueName: \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.776562 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.776510 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.776562 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:32.776514 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-serving-cert: secret "maas-api-e2e-api-a-08eeaa-serving-cert" not found Nov 02 20:11:32.776642 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:32.776579 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls podName:9f6ae38b-f81e-40dd-8d62-97fd6ab77401 nodeName:}" failed. No retries permitted until 2026-11-02 20:11:33.276565748 +0000 UTC m=+1376.883110223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls") pod "maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" (UID: "9f6ae38b-f81e-40dd-8d62-97fd6ab77401") : secret "maas-api-e2e-api-a-08eeaa-serving-cert" not found Nov 02 20:11:32.777161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.777144 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.778811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.778795 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:32.865807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:32.864595 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58m48\" (UniqueName: \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:33.294059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:33.294028 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:33.296504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:33.296476 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") pod \"maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:33.473110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:33.473081 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:33.839227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:33.839178 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:11:33.844405 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:11:33.844362 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6ae38b_f81e_40dd_8d62_97fd6ab77401.slice/crio-64721f6d55968480dd2516f7706c0b9b4f27ba2e0fe6ab9e17cd7276364ec985 WatchSource:0}: Error finding container 64721f6d55968480dd2516f7706c0b9b4f27ba2e0fe6ab9e17cd7276364ec985: Status 404 returned error can't find the container with id 64721f6d55968480dd2516f7706c0b9b4f27ba2e0fe6ab9e17cd7276364ec985 Nov 02 20:11:34.180949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:34.180862 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" event={"ID":"9f6ae38b-f81e-40dd-8d62-97fd6ab77401","Type":"ContainerStarted","Data":"64721f6d55968480dd2516f7706c0b9b4f27ba2e0fe6ab9e17cd7276364ec985"} Nov 02 20:11:35.188259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:35.188175 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" event={"ID":"9f6ae38b-f81e-40dd-8d62-97fd6ab77401","Type":"ContainerStarted","Data":"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978"} Nov 02 20:11:35.188611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:35.188266 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:35.298069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:35.298018 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" podStartSLOduration=2.443053931 podStartE2EDuration="3.298002412s" totalImagesPullingTime="854.948481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:35.295918178 +0000 UTC m=+1378.902462689" watchObservedRunningTime="2026-11-02 20:11:35.298002412 +0000 UTC m=+1378.904546954" Nov 02 20:11:41.199022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:41.198992 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:11:48.450977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.450942 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:11:48.455849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.455830 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.554135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.554012 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:11:48.556432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.556403 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.556609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.556590 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2sk\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.556772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.556753 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.556999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.556982 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.557184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.557169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.557316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.557302 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.557440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.557426 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.557544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.557531 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.557718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.557702 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.599240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.599206 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-671e7c-openshift-default-dockercfg-ctrxx\"" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.664657 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.664826 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.664888 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.664946 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.664994 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.665278 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.665355 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.665381 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2sk\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.665626 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.671810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.666184 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.705808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.697955 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.706571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.706538 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.707131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.707111 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.707953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.707931 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.725334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.725303 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.753757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.728240 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.790255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.789692 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2sk\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:48.796705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:48.796665 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-674b6\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:49.069541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:49.069456 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:49.485263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:49.485239 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:11:49.486115 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:11:49.486077 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8a815a_46f5_4d8d_aaef_3228369bbbef.slice/crio-a82f938d70a3aa8a8216ebb26792ed81912d09fd9e2f420e520bed094c9112ef WatchSource:0}: Error finding container a82f938d70a3aa8a8216ebb26792ed81912d09fd9e2f420e520bed094c9112ef: Status 404 returned error can't find the container with id a82f938d70a3aa8a8216ebb26792ed81912d09fd9e2f420e520bed094c9112ef Nov 02 20:11:49.488683 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:49.488650 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:49.488787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:49.488714 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:49.488787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:49.488762 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:50.293389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:50.293354 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerStarted","Data":"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5"} Nov 02 20:11:50.293595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:50.293407 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerStarted","Data":"a82f938d70a3aa8a8216ebb26792ed81912d09fd9e2f420e520bed094c9112ef"} Nov 02 20:11:51.024266 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:11:51.024230 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:11:51.070096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:51.070068 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:51.075113 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:51.075092 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:51.261262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:51.261212 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podStartSLOduration=3.2611991 podStartE2EDuration="3.2611991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:50.431369922 +0000 UTC m=+1394.037914420" watchObservedRunningTime="2026-11-02 20:11:51.2611991 +0000 UTC m=+1394.867743597" Nov 02 20:11:51.317260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:51.317153 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:11:51.318430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:11:51.318398 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:12:01.387490 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.387456 5136 generic.go:356] "Generic (PLEG): container finished" podID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerID="eb6ca35d90f557548174c988dbcca9da5326465deba04d0f9a04b9fe8cf22f80" exitCode=0 Nov 02 20:12:01.388044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.387526 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerDied","Data":"eb6ca35d90f557548174c988dbcca9da5326465deba04d0f9a04b9fe8cf22f80"} Nov 02 20:12:01.388117 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.388065 5136 scope.go:122] "RemoveContainer" containerID="eb6ca35d90f557548174c988dbcca9da5326465deba04d0f9a04b9fe8cf22f80" Nov 02 20:12:01.389893 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.389859 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:01.390015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.389945 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:01.390015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:01.389991 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.418928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:02.418892 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerStarted","Data":"ef374d182ac03a16f3042909d53561b29fdf10699445b76c15dcc209f68d185d"} Nov 02 20:12:02.512956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:02.512916 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:12:02.514642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:02.514616 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:02.514799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:02.514664 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:03.513153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:03.513112 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:03.513644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:03.513176 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:04.513079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:04.513043 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:04.513280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:04.513185 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:05.513059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:05.513025 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:05.513269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:05.513077 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:05.709754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:05.709545 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:12:06.513321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:06.513150 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:06.513321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:06.513212 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:07.513920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:07.513885 5136 patch_prober.go:29] interesting pod/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:12:07.514337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:07.513947 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:12:08.528391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:08.528358 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:12:08.531087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:08.531064 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:12:16.580091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.580057 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:12:16.584610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.584591 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.663779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.657241 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:12:16.717767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.716521 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-341fb2-openshift-default-dockercfg-wgln9\"" Nov 02 20:12:16.771132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.735849 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.774756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.771508 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8xz\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.778767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.775016 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.787761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.778984 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.840976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.791895 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.840976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.814870 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.840976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.814994 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.840976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.815060 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.840976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.815083 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916522 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916586 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8xz\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916639 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916673 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916774 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916827 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916915 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.916981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.916976 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.917195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.917003 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.917686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.917658 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.917885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.917861 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.936761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.918143 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.936761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.918725 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.936761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.919008 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.936761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.924809 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.936761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.928374 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.998557 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.995936 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:16.998936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:16.997087 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8xz\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\") pod \"e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:17.201806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:17.199024 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:17.670603 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:12:17.670570 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33efceeb_0fd9_43f5_8245_2672bfc65adc.slice/crio-f8f172e558ef28fb8d6c4b79026261ddb35b25f3fe978635a066215e1413c968 WatchSource:0}: Error finding container f8f172e558ef28fb8d6c4b79026261ddb35b25f3fe978635a066215e1413c968: Status 404 returned error can't find the container with id f8f172e558ef28fb8d6c4b79026261ddb35b25f3fe978635a066215e1413c968 Nov 02 20:12:17.670979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:17.670634 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:12:17.673219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:17.673189 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:17.673300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:17.673252 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:17.673300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:17.673280 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:18.536791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:18.536757 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerStarted","Data":"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20"} Nov 02 20:12:18.536974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:18.536815 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerStarted","Data":"f8f172e558ef28fb8d6c4b79026261ddb35b25f3fe978635a066215e1413c968"} Nov 02 20:12:18.694155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:18.694102 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podStartSLOduration=2.694087846 podStartE2EDuration="2.694087846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:18.693598318 +0000 UTC m=+1422.300142815" watchObservedRunningTime="2026-11-02 20:12:18.694087846 +0000 UTC m=+1422.300632343" Nov 02 20:12:19.200318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:19.200281 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:19.205947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:19.205920 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:19.567159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:19.565805 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:19.568462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:19.568416 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:12:21.007986 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:12:21.007950 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:12:40.778752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.777774 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:12:40.837964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.837924 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:12:40.838196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.838061 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:40.891833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.891806 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-671e7c-serving-cert\"" Nov 02 20:12:40.899567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.899539 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:40.899700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.899651 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:40.899700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.899682 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:40.899797 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:40.899779 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrnj\" (UniqueName: \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.000923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.000890 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.001098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.001004 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.001098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.001037 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.001098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.001085 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrnj\" (UniqueName: \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.001278 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:12:41.001197 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-serving-cert: secret "maas-api-e2e-api-a-671e7c-serving-cert" not found Nov 02 20:12:41.001278 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:12:41.001272 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls podName:f3563925-8b17-4e84-a9ee-0644dbdbde48 nodeName:}" failed. No retries permitted until 2026-11-02 20:12:41.501255784 +0000 UTC m=+1445.107800297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls") pod "maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" (UID: "f3563925-8b17-4e84-a9ee-0644dbdbde48") : secret "maas-api-e2e-api-a-671e7c-serving-cert" not found Nov 02 20:12:41.001636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.001614 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.003404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.003381 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.055523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.055471 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrnj\" (UniqueName: \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.517682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.517643 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.520019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.519997 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") pod \"maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:41.755496 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:41.755468 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:42.126241 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:42.124691 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:12:42.129981 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:12:42.129953 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3563925_8b17_4e84_a9ee_0644dbdbde48.slice/crio-a7cb06bbf191eff674558df7b37346fa85e20d16b2c23639eeb30d3abec4e190 WatchSource:0}: Error finding container a7cb06bbf191eff674558df7b37346fa85e20d16b2c23639eeb30d3abec4e190: Status 404 returned error can't find the container with id a7cb06bbf191eff674558df7b37346fa85e20d16b2c23639eeb30d3abec4e190 Nov 02 20:12:42.730656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:42.730625 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" event={"ID":"f3563925-8b17-4e84-a9ee-0644dbdbde48","Type":"ContainerStarted","Data":"a7cb06bbf191eff674558df7b37346fa85e20d16b2c23639eeb30d3abec4e190"} Nov 02 20:12:43.738740 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:43.738701 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" event={"ID":"f3563925-8b17-4e84-a9ee-0644dbdbde48","Type":"ContainerStarted","Data":"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7"} Nov 02 20:12:43.739139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:43.738796 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:43.848778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:43.848679 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" podStartSLOduration=2.941855706 podStartE2EDuration="3.848663142s" totalImagesPullingTime="906.807436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:43.846874737 +0000 UTC m=+1447.453419235" watchObservedRunningTime="2026-11-02 20:12:43.848663142 +0000 UTC m=+1447.455207641" Nov 02 20:12:49.750442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:12:49.750360 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:12:51.004701 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:12:51.004668 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:13:06.319273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.319192 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:06.319273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.319253 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:06.990180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.990145 5136 generic.go:356] "Generic (PLEG): container finished" podID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerID="ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5" exitCode=0 Nov 02 20:13:06.990344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.990215 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerDied","Data":"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5"} Nov 02 20:13:06.990659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.990642 5136 scope.go:122] "RemoveContainer" containerID="ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5" Nov 02 20:13:06.992223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.992196 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:06.992290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.992268 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:06.992326 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:06.992301 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:08.027695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:08.027653 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerStarted","Data":"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4"} Nov 02 20:13:08.070284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:08.070258 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:13:08.070469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:08.070448 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:08.070521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:08.070504 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:09.070661 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:09.070632 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:09.071081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:09.070683 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:10.070365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:10.070332 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:10.070551 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:10.070388 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:11.069907 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:11.069870 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:11.070300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:11.069924 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:12.070102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:12.070069 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:12.070568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:12.070129 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:13.070558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:13.070518 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:13.071074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:13.070597 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:14.073920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:14.073894 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:13:14.074324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:14.074301 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:13:14.074825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:14.074809 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:13:16.466525 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.466489 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:13:16.470997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.470975 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.527095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.527052 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:13:16.559764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.559714 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.559921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.559886 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.559965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.559939 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.560074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.560061 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.567302 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.567282 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-341fb2-serving-cert\"" Nov 02 20:13:16.661005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.660975 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.661179 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.661076 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.661179 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.661133 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.661298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.661180 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.661351 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:16.661315 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-serving-cert: secret "maas-api-e2e-api-b-341fb2-serving-cert" not found Nov 02 20:13:16.661397 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:16.661368 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls podName:f2bef18d-ab8d-48b5-af70-8d0d7bd965bb nodeName:}" failed. No retries permitted until 2026-11-02 20:13:17.161350966 +0000 UTC m=+1480.767895456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls") pod "maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb") : secret "maas-api-e2e-api-b-341fb2-serving-cert" not found Nov 02 20:13:16.661851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.661823 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.663627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.663604 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:16.715884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:16.715853 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:17.172652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:17.172619 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:17.175154 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:17.175131 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") pod \"maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:17.383646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:17.383609 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:17.751646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:17.751583 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:13:17.752851 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:13:17.752794 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bef18d_ab8d_48b5_af70_8d0d7bd965bb.slice/crio-089211336d1520d3a08ac65e9d82db717768dd1f390f288d0fb1ed6f4a8b35b8 WatchSource:0}: Error finding container 089211336d1520d3a08ac65e9d82db717768dd1f390f288d0fb1ed6f4a8b35b8: Status 404 returned error can't find the container with id 089211336d1520d3a08ac65e9d82db717768dd1f390f288d0fb1ed6f4a8b35b8 Nov 02 20:13:18.093721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:18.093636 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" event={"ID":"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb","Type":"ContainerStarted","Data":"089211336d1520d3a08ac65e9d82db717768dd1f390f288d0fb1ed6f4a8b35b8"} Nov 02 20:13:19.101560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:19.101530 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" event={"ID":"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb","Type":"ContainerStarted","Data":"57cc8aa65f1c3597c2c241bbdf461a59d8ceefccc2f07aaacd3a0ae9b979e5e6"} Nov 02 20:13:19.102021 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:19.101667 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:19.212438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:19.212389 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" podStartSLOduration=2.282236125 podStartE2EDuration="3.21237525s" totalImagesPullingTime="930.139125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.209640373 +0000 UTC m=+1482.816184873" watchObservedRunningTime="2026-11-02 20:13:19.21237525 +0000 UTC m=+1482.818919788" Nov 02 20:13:21.004451 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:21.004416 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:13:25.113521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:25.113490 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:13:31.256871 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.256829 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:13:31.260615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.260589 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.314136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.314101 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:13:31.356528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.356495 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a3021d-serving-cert\"" Nov 02 20:13:31.396685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.396644 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.396933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.396902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.397176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.397147 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2dh\" (UniqueName: \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.397224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.397209 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.498861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.498830 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.499263 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:31.499170 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-serving-cert: secret "maas-api-e2e-api-b-a3021d-serving-cert" not found Nov 02 20:13:31.499572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.499380 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2dh\" (UniqueName: \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.499831 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:31.499810 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls podName:55ca8acc-6a45-458e-be14-cc112e7d9e9f nodeName:}" failed. No retries permitted until 2026-11-02 20:13:31.99952697 +0000 UTC m=+1495.606071461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls") pod "maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" (UID: "55ca8acc-6a45-458e-be14-cc112e7d9e9f") : secret "maas-api-e2e-api-b-a3021d-serving-cert" not found Nov 02 20:13:31.500016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.499882 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.500016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.499974 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.500696 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.500668 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.516036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.503764 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:31.575630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:31.575596 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2dh\" (UniqueName: \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:32.015597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:32.015564 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:32.018043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:32.018015 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") pod \"maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:32.174317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:32.174279 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:32.547264 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:13:32.547236 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ca8acc_6a45_458e_be14_cc112e7d9e9f.slice/crio-090ebbb5324c5c8090bbfae102dfc14843a071ef2c19841580c8cfed08bb0396 WatchSource:0}: Error finding container 090ebbb5324c5c8090bbfae102dfc14843a071ef2c19841580c8cfed08bb0396: Status 404 returned error can't find the container with id 090ebbb5324c5c8090bbfae102dfc14843a071ef2c19841580c8cfed08bb0396 Nov 02 20:13:32.550387 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:32.550364 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:13:33.198203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:33.198163 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" event={"ID":"55ca8acc-6a45-458e-be14-cc112e7d9e9f","Type":"ContainerStarted","Data":"090ebbb5324c5c8090bbfae102dfc14843a071ef2c19841580c8cfed08bb0396"} Nov 02 20:13:34.207392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:34.207352 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" event={"ID":"55ca8acc-6a45-458e-be14-cc112e7d9e9f","Type":"ContainerStarted","Data":"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b"} Nov 02 20:13:34.207836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:34.207569 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:34.319895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:34.319840 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" podStartSLOduration=2.386257467 podStartE2EDuration="3.319826955s" totalImagesPullingTime="933.569488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:34.316833304 +0000 UTC m=+1497.923377804" watchObservedRunningTime="2026-11-02 20:13:34.319826955 +0000 UTC m=+1497.926371453" Nov 02 20:13:40.218861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:13:40.218830 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:13:51.005125 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:13:51.005086 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:14:01.802841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.802805 5136 generic.go:356] "Generic (PLEG): container finished" podID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerID="c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20" exitCode=0 Nov 02 20:14:01.803280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.802870 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerDied","Data":"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20"} Nov 02 20:14:01.803280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.803264 5136 scope.go:122] "RemoveContainer" containerID="c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20" Nov 02 20:14:01.804937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.804905 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:01.805008 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.804980 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:01.805045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:01.805017 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:02.811309 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:02.811274 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerStarted","Data":"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f"} Nov 02 20:14:03.202749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:03.199639 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:14:03.211564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:03.209861 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:03.211564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:03.209940 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:04.206176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:04.206137 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:04.206815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:04.206781 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:04.563338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:04.563259 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:14:05.201762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:05.201104 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:05.201762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:05.201167 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:06.208333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:06.208288 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:06.208845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:06.208355 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:07.213755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:07.213271 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:07.213755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:07.213332 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:08.207899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:08.207862 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:14:08.208208 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:08.208179 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:14:09.204870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:09.204835 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:14:09.206159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:09.206136 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:14:21.003845 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:14:21.003757 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:14:45.760408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.760374 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:14:45.764148 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.764118 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.792386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.792357 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf92\" (UniqueName: \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.792512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.792401 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.792512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.792416 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.792601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.792571 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.814981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.814959 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-15203f-serving-cert\"" Nov 02 20:14:45.818428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.818393 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:14:45.893406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.893374 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.893586 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:14:45.893503 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-serving-cert: secret "maas-api-e2e-api-a-15203f-serving-cert" not found Nov 02 20:14:45.893586 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:14:45.893570 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls podName:f73340a0-5315-462e-b5fa-ce602d9894ea nodeName:}" failed. No retries permitted until 2026-11-02 20:14:46.393553992 +0000 UTC m=+1570.000098488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls") pod "maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" (UID: "f73340a0-5315-462e-b5fa-ce602d9894ea") : secret "maas-api-e2e-api-a-15203f-serving-cert" not found Nov 02 20:14:45.893685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.893628 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf92\" (UniqueName: \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.893685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.893658 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.893685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.893673 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.894342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.894321 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.896187 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.896167 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:45.949878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:45.949851 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf92\" (UniqueName: \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:46.414513 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:46.414477 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:46.416970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:46.416942 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") pod \"maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:46.677082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:46.676999 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:47.047116 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:47.047088 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:14:47.047583 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:14:47.047422 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73340a0_5315_462e_b5fa_ce602d9894ea.slice/crio-717b4c81e497acf1f183e8a3a7a116581ac12bfe7c0be5b1b0b7bbe3f1c1fd98 WatchSource:0}: Error finding container 717b4c81e497acf1f183e8a3a7a116581ac12bfe7c0be5b1b0b7bbe3f1c1fd98: Status 404 returned error can't find the container with id 717b4c81e497acf1f183e8a3a7a116581ac12bfe7c0be5b1b0b7bbe3f1c1fd98 Nov 02 20:14:47.130621 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:47.130583 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" event={"ID":"f73340a0-5315-462e-b5fa-ce602d9894ea","Type":"ContainerStarted","Data":"717b4c81e497acf1f183e8a3a7a116581ac12bfe7c0be5b1b0b7bbe3f1c1fd98"} Nov 02 20:14:48.143721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:48.143273 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" event={"ID":"f73340a0-5315-462e-b5fa-ce602d9894ea","Type":"ContainerStarted","Data":"a26272eaf53c91469cf0d01488d153e8979a02cdaec1af60b3186f4d005d5c04"} Nov 02 20:14:48.144302 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:48.144280 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:14:48.281898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:48.281810 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" podStartSLOduration=2.364845775 podStartE2EDuration="3.281797037s" totalImagesPullingTime="916.951262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:48.281460759 +0000 UTC m=+1571.888005251" watchObservedRunningTime="2026-11-02 20:14:48.281797037 +0000 UTC m=+1571.888341534" Nov 02 20:14:51.006663 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:14:51.006611 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:14:55.173498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:14:55.173470 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:15:02.146972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.146933 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:15:02.150936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.150913 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.197459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.197422 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.197459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.197464 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.197672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.197573 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xgx\" (UniqueName: \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.197795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.197778 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.203990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.203960 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:15:02.247106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.247076 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e0aa5c-serving-cert\"" Nov 02 20:15:02.305273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.305232 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.305464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.305292 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.305464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.305373 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xgx\" (UniqueName: \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.305592 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:02.305572 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-serving-cert: secret "maas-api-e2e-api-b-e0aa5c-serving-cert" not found Nov 02 20:15:02.305667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.305646 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.305667 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:02.305660 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls podName:c05920b8-5726-48d2-b219-bdd3b213a039 nodeName:}" failed. No retries permitted until 2026-11-02 20:15:02.805640378 +0000 UTC m=+1586.412184903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls") pod "maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" (UID: "c05920b8-5726-48d2-b219-bdd3b213a039") : secret "maas-api-e2e-api-b-e0aa5c-serving-cert" not found Nov 02 20:15:02.306668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.306640 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.318249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.318225 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.359590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.359558 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xgx\" (UniqueName: \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.821562 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.821525 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:02.824186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:02.824159 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") pod \"maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:03.063925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:03.063889 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:03.434445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:03.434403 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:15:03.435222 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:15:03.435193 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05920b8_5726_48d2_b219_bdd3b213a039.slice/crio-4b68515ca6e4bb8c11e89526c42e93be65a63ec1959c32c7f9e98c9be13b60a5 WatchSource:0}: Error finding container 4b68515ca6e4bb8c11e89526c42e93be65a63ec1959c32c7f9e98c9be13b60a5: Status 404 returned error can't find the container with id 4b68515ca6e4bb8c11e89526c42e93be65a63ec1959c32c7f9e98c9be13b60a5 Nov 02 20:15:04.291594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:04.291550 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" event={"ID":"c05920b8-5726-48d2-b219-bdd3b213a039","Type":"ContainerStarted","Data":"4b68515ca6e4bb8c11e89526c42e93be65a63ec1959c32c7f9e98c9be13b60a5"} Nov 02 20:15:05.299445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:05.299403 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" event={"ID":"c05920b8-5726-48d2-b219-bdd3b213a039","Type":"ContainerStarted","Data":"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc"} Nov 02 20:15:05.299956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:05.299589 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:05.411367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:05.411316 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" podStartSLOduration=2.55278495 podStartE2EDuration="3.411303538s" totalImagesPullingTime="858.518588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:05.410790164 +0000 UTC m=+1589.017334662" watchObservedRunningTime="2026-11-02 20:15:05.411303538 +0000 UTC m=+1589.017848101" Nov 02 20:15:07.124237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.124198 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:15:07.124689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.124407 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" podUID="822a4355-c346-4689-bfe2-21344dedf120" containerName="maas-api" containerID="cri-o://c68cc10a47cb84ab21764fd2fef2cf5624757bc9e4529abdfed2b0112d46ab2b" gracePeriod=30 Nov 02 20:15:07.332706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.332668 5136 generic.go:356] "Generic (PLEG): container finished" podID="822a4355-c346-4689-bfe2-21344dedf120" containerID="c68cc10a47cb84ab21764fd2fef2cf5624757bc9e4529abdfed2b0112d46ab2b" exitCode=0 Nov 02 20:15:07.333442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.333398 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" event={"ID":"822a4355-c346-4689-bfe2-21344dedf120","Type":"ContainerDied","Data":"c68cc10a47cb84ab21764fd2fef2cf5624757bc9e4529abdfed2b0112d46ab2b"} Nov 02 20:15:07.588503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.588464 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:15:07.711693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.709501 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") pod \"822a4355-c346-4689-bfe2-21344dedf120\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " Nov 02 20:15:07.711693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.709589 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\") pod \"822a4355-c346-4689-bfe2-21344dedf120\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " Nov 02 20:15:07.711693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.709678 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\" (UniqueName: \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\") pod \"822a4355-c346-4689-bfe2-21344dedf120\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " Nov 02 20:15:07.711693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.709903 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\") pod \"822a4355-c346-4689-bfe2-21344dedf120\" (UID: \"822a4355-c346-4689-bfe2-21344dedf120\") " Nov 02 20:15:07.720783 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.716206 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca" pod "822a4355-c346-4689-bfe2-21344dedf120" (UID: "822a4355-c346-4689-bfe2-21344dedf120"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:07.720783 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.719372 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls" pod "822a4355-c346-4689-bfe2-21344dedf120" (UID: "822a4355-c346-4689-bfe2-21344dedf120"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:07.725842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.725789 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl" pod "822a4355-c346-4689-bfe2-21344dedf120" (UID: "822a4355-c346-4689-bfe2-21344dedf120"). InnerVolumeSpecName "kube-api-access-4t2tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:07.725969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.725842 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs" pod "822a4355-c346-4689-bfe2-21344dedf120" (UID: "822a4355-c346-4689-bfe2-21344dedf120"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:07.812725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.812685 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822a4355-c346-4689-bfe2-21344dedf120-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:07.812725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.812726 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2tl\" (UniqueName: \"kubernetes.io/projected/822a4355-c346-4689-bfe2-21344dedf120-kube-api-access-4t2tl\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:07.812957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.812757 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:07.812957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:07.812772 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822a4355-c346-4689-bfe2-21344dedf120-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:08.340859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:08.340832 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" Nov 02 20:15:08.340859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:08.340849 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv" event={"ID":"822a4355-c346-4689-bfe2-21344dedf120","Type":"ContainerDied","Data":"eec046031cc755dd9b41eba17dc4dbc636e051e3b041542ca5c2646fd3da1bf0"} Nov 02 20:15:08.341380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:08.340936 5136 scope.go:122] "RemoveContainer" containerID="c68cc10a47cb84ab21764fd2fef2cf5624757bc9e4529abdfed2b0112d46ab2b" Nov 02 20:15:08.497717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:08.497688 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:15:08.548559 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:08.548529 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-45c182-5765588bbd-97ltv"] Nov 02 20:15:10.547837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:10.547797 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822a4355-c346-4689-bfe2-21344dedf120" path="/var/lib/kubelet/pods/822a4355-c346-4689-bfe2-21344dedf120/volumes" Nov 02 20:15:11.312606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:11.312580 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:15:18.904942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:18.904908 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:15:18.905359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:18.905117 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" podUID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" containerName="maas-api" containerID="cri-o://b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978" gracePeriod=30 Nov 02 20:15:19.395698 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.395336 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:15:19.477219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.477184 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") pod \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " Nov 02 20:15:19.477497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.477481 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\") pod \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " Nov 02 20:15:19.477775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.477756 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\") pod \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " Nov 02 20:15:19.478090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.478070 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\" (UniqueName: \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\") pod \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\" (UID: \"9f6ae38b-f81e-40dd-8d62-97fd6ab77401\") " Nov 02 20:15:19.481381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.481357 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca" pod "9f6ae38b-f81e-40dd-8d62-97fd6ab77401" (UID: "9f6ae38b-f81e-40dd-8d62-97fd6ab77401"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:19.497120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.497080 5136 generic.go:356] "Generic (PLEG): container finished" podID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" containerID="b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978" exitCode=0 Nov 02 20:15:19.497375 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.497350 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" event={"ID":"9f6ae38b-f81e-40dd-8d62-97fd6ab77401","Type":"ContainerDied","Data":"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978"} Nov 02 20:15:19.497541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.497521 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" event={"ID":"9f6ae38b-f81e-40dd-8d62-97fd6ab77401","Type":"ContainerDied","Data":"64721f6d55968480dd2516f7706c0b9b4f27ba2e0fe6ab9e17cd7276364ec985"} Nov 02 20:15:19.498428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.497845 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm" Nov 02 20:15:19.498919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.497722 5136 scope.go:122] "RemoveContainer" containerID="b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978" Nov 02 20:15:19.503749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.503694 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48" pod "9f6ae38b-f81e-40dd-8d62-97fd6ab77401" (UID: "9f6ae38b-f81e-40dd-8d62-97fd6ab77401"). InnerVolumeSpecName "kube-api-access-58m48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:19.506545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.506503 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs" pod "9f6ae38b-f81e-40dd-8d62-97fd6ab77401" (UID: "9f6ae38b-f81e-40dd-8d62-97fd6ab77401"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:19.507095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.507061 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls" pod "9f6ae38b-f81e-40dd-8d62-97fd6ab77401" (UID: "9f6ae38b-f81e-40dd-8d62-97fd6ab77401"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:19.544477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.544448 5136 scope.go:122] "RemoveContainer" containerID="b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978" Nov 02 20:15:19.544904 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:19.544876 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978\": container with ID starting with b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978 not found: ID does not exist" containerID="b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978" Nov 02 20:15:19.545002 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.544920 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978"} err="failed to get container status \"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978\": rpc error: code = NotFound desc = could not find container \"b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978\": container with ID starting with b827ab5f237f99f010dc056429eab621ba6efd346db36c3488335ff7dbe8d978 not found: ID does not exist" Nov 02 20:15:19.580013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.579980 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58m48\" (UniqueName: \"kubernetes.io/projected/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-kube-api-access-58m48\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.580013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.580008 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.580213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.580023 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.580213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.580036 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6ae38b-f81e-40dd-8d62-97fd6ab77401-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.915307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.915277 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:15:19.962317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:19.962282 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08eeaa-766c858d85-n9xmm"] Nov 02 20:15:20.547254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:20.547220 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" path="/var/lib/kubelet/pods/9f6ae38b-f81e-40dd-8d62-97fd6ab77401/volumes" Nov 02 20:15:21.007119 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:21.007084 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:15:21.625282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.625243 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:15:21.625569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.625534 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" containerID="cri-o://35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6" gracePeriod=30 Nov 02 20:15:21.673148 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.673115 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:15:21.673368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.673344 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" containerID="cri-o://395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17" gracePeriod=30 Nov 02 20:15:21.676327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.676300 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:15:21.676553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:21.676528 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" containerID="cri-o://c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726" gracePeriod=30 Nov 02 20:15:26.374833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:26.374800 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:15:26.375338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:26.375076 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" containerID="cri-o://ef374d182ac03a16f3042909d53561b29fdf10699445b76c15dcc209f68d185d" gracePeriod=30 Nov 02 20:15:26.982400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:26.982369 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:15:27.013898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.013876 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:15:27.040581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.040560 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:15:27.138104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138068 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.138288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138115 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.138288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138142 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.138288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138190 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.138473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138394 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.138589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138561 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:27.138706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138586 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.138706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138615 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.138706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138662 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.138706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138700 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.138975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138766 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.138975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138816 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.138975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138855 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.138975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138937 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.138978 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139009 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139062 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139106 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139160 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139199 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139239 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139274 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139312 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139352 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139387 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139438 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\") pod \"1f166cc2-1183-4da4-94c3-6fcb437fbade\" (UID: \"1f166cc2-1183-4da4-94c3-6fcb437fbade\") " Nov 02 20:15:27.139497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139479 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139547 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\") pod \"6194de7e-caca-4b06-a83a-6a6a492b6bca\" (UID: \"6194de7e-caca-4b06-a83a-6a6a492b6bca\") " Nov 02 20:15:27.139869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139626 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.139869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139675 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\") pod \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\" (UID: \"eb5feec8-d535-4a7e-a1ea-33999b4f9978\") " Nov 02 20:15:27.140018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.139887 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.140581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.140557 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.140677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.140587 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f166cc2-1183-4da4-94c3-6fcb437fbade-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.140677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.140604 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.140677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.140619 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.140677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.140656 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.141051 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141025 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.141124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141095 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:27.141390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141367 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.141462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141412 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141650 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141867 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141930 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.141931 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.142152 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.142166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.142382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.142168 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.144612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.144586 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.145713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.145643 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.145713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.145678 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:27.145906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.145886 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:27.146053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146030 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "kube-api-access-gdxf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.146122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146040 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:27.146167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146123 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.146304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146284 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj" pod "6194de7e-caca-4b06-a83a-6a6a492b6bca" (UID: "6194de7e-caca-4b06-a83a-6a6a492b6bca"). InnerVolumeSpecName "kube-api-access-mz4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.146393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146306 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "kube-api-access-qsqf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.146556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.146535 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo" pod "1f166cc2-1183-4da4-94c3-6fcb437fbade" (UID: "1f166cc2-1183-4da4-94c3-6fcb437fbade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:27.148082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.148059 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token" pod "eb5feec8-d535-4a7e-a1ea-33999b4f9978" (UID: "eb5feec8-d535-4a7e-a1ea-33999b4f9978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:27.242103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242066 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242104 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxf8\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-kube-api-access-gdxf8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242116 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242127 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242141 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242151 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242160 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242168 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242176 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242185 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6194de7e-caca-4b06-a83a-6a6a492b6bca-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242195 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242203 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242211 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242219 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242227 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6194de7e-caca-4b06-a83a-6a6a492b6bca-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242235 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242244 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242252 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5feec8-d535-4a7e-a1ea-33999b4f9978-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242260 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4hj\" (UniqueName: \"kubernetes.io/projected/6194de7e-caca-4b06-a83a-6a6a492b6bca-kube-api-access-mz4hj\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242268 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242277 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f166cc2-1183-4da4-94c3-6fcb437fbade-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242285 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqf4\" (UniqueName: \"kubernetes.io/projected/eb5feec8-d535-4a7e-a1ea-33999b4f9978-kube-api-access-qsqf4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.242317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.242293 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6194de7e-caca-4b06-a83a-6a6a492b6bca-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:27.604622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.604536 5136 generic.go:356] "Generic (PLEG): container finished" podID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerID="395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17" exitCode=0 Nov 02 20:15:27.604622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.604588 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" Nov 02 20:15:27.605167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.604619 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" event={"ID":"1f166cc2-1183-4da4-94c3-6fcb437fbade","Type":"ContainerDied","Data":"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17"} Nov 02 20:15:27.605167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.604668 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m" event={"ID":"1f166cc2-1183-4da4-94c3-6fcb437fbade","Type":"ContainerDied","Data":"d5a0fe1a919a262cb600d12d8ad5ee3961b831c39d78bc2fd47f2c86a88d494a"} Nov 02 20:15:27.605167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.604689 5136 scope.go:122] "RemoveContainer" containerID="395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17" Nov 02 20:15:27.606142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.606122 5136 generic.go:356] "Generic (PLEG): container finished" podID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerID="35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6" exitCode=0 Nov 02 20:15:27.606244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.606169 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" Nov 02 20:15:27.606244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.606201 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" event={"ID":"eb5feec8-d535-4a7e-a1ea-33999b4f9978","Type":"ContainerDied","Data":"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6"} Nov 02 20:15:27.606367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.606254 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" event={"ID":"eb5feec8-d535-4a7e-a1ea-33999b4f9978","Type":"ContainerDied","Data":"aef9e2ad522eb471faa8423b0e7f98ebbd79cc2b9fdaa6a593ef5409c0c04735"} Nov 02 20:15:27.607800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.607777 5136 generic.go:356] "Generic (PLEG): container finished" podID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerID="c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726" exitCode=0 Nov 02 20:15:27.607932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.607839 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" Nov 02 20:15:27.607932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.607857 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" event={"ID":"6194de7e-caca-4b06-a83a-6a6a492b6bca","Type":"ContainerDied","Data":"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726"} Nov 02 20:15:27.607932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.607909 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8" event={"ID":"6194de7e-caca-4b06-a83a-6a6a492b6bca","Type":"ContainerDied","Data":"303481d4de85d7f2b416f34674dc0a8b557d1a4ebf1688e79359369f13030a63"} Nov 02 20:15:27.617423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.617402 5136 scope.go:122] "RemoveContainer" containerID="395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17" Nov 02 20:15:27.617724 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:27.617705 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17\": container with ID starting with 395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17 not found: ID does not exist" containerID="395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17" Nov 02 20:15:27.617802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.617761 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17"} err="failed to get container status \"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17\": rpc error: code = NotFound desc = could not find container \"395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17\": container with ID starting with 395eed78a2ccd3f10e720d5d42b408bd6046f64b771127f2426624ff365cbb17 not found: ID does not exist" Nov 02 20:15:27.617802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.617786 5136 scope.go:122] "RemoveContainer" containerID="35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6" Nov 02 20:15:27.627210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.627192 5136 scope.go:122] "RemoveContainer" containerID="35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6" Nov 02 20:15:27.627475 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:27.627452 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6\": container with ID starting with 35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6 not found: ID does not exist" containerID="35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6" Nov 02 20:15:27.627580 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.627481 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6"} err="failed to get container status \"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6\": rpc error: code = NotFound desc = could not find container \"35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6\": container with ID starting with 35f05898dd627e54ec0f44dcd9b5f0ae4898b92592775acdfb4cfdba359e26e6 not found: ID does not exist" Nov 02 20:15:27.627580 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.627499 5136 scope.go:122] "RemoveContainer" containerID="c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726" Nov 02 20:15:27.636212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.636195 5136 scope.go:122] "RemoveContainer" containerID="c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726" Nov 02 20:15:27.636440 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:27.636422 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726\": container with ID starting with c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726 not found: ID does not exist" containerID="c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726" Nov 02 20:15:27.636500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.636449 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726"} err="failed to get container status \"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726\": rpc error: code = NotFound desc = could not find container \"c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726\": container with ID starting with c6fe41b51232bcebe0185b28d088a76fc598536175f8b5a3165bb9b864082726 not found: ID does not exist" Nov 02 20:15:27.761406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.761372 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:15:27.765428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.765406 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-bhs7m"] Nov 02 20:15:27.869839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.869757 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:15:27.873087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.873062 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng"] Nov 02 20:15:27.976218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.976173 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:27.976413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.976259 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-xpvng" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:27.976413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.976189 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:15:27.980306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:27.980282 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-px9p8"] Nov 02 20:15:28.564925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:28.562376 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" path="/var/lib/kubelet/pods/1f166cc2-1183-4da4-94c3-6fcb437fbade/volumes" Nov 02 20:15:28.564925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:28.563246 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" path="/var/lib/kubelet/pods/6194de7e-caca-4b06-a83a-6a6a492b6bca/volumes" Nov 02 20:15:28.564925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:28.564038 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" path="/var/lib/kubelet/pods/eb5feec8-d535-4a7e-a1ea-33999b4f9978/volumes" Nov 02 20:15:31.016062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.016026 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:15:31.016493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.016243 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" podUID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" containerName="maas-api" containerID="cri-o://2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479" gracePeriod=30 Nov 02 20:15:31.356701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.356674 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:15:31.536752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.531994 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\") pod \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " Nov 02 20:15:31.536752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.532173 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\") pod \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " Nov 02 20:15:31.536752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.532214 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") pod \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " Nov 02 20:15:31.536752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.532278 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\" (UniqueName: \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\") pod \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\" (UID: \"496d118d-3398-4f92-aa8c-d8cc19c77d3b\") " Nov 02 20:15:31.536752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.534119 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca" pod "496d118d-3398-4f92-aa8c-d8cc19c77d3b" (UID: "496d118d-3398-4f92-aa8c-d8cc19c77d3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:31.583290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.583238 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4" pod "496d118d-3398-4f92-aa8c-d8cc19c77d3b" (UID: "496d118d-3398-4f92-aa8c-d8cc19c77d3b"). InnerVolumeSpecName "kube-api-access-9tpn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:31.584686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.584649 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls" pod "496d118d-3398-4f92-aa8c-d8cc19c77d3b" (UID: "496d118d-3398-4f92-aa8c-d8cc19c77d3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:31.585184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.585164 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs" pod "496d118d-3398-4f92-aa8c-d8cc19c77d3b" (UID: "496d118d-3398-4f92-aa8c-d8cc19c77d3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:31.634389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.634355 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d118d-3398-4f92-aa8c-d8cc19c77d3b-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:31.634389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.634387 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:31.634616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.634400 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpn4\" (UniqueName: \"kubernetes.io/projected/496d118d-3398-4f92-aa8c-d8cc19c77d3b-kube-api-access-9tpn4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:31.634616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.634415 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d118d-3398-4f92-aa8c-d8cc19c77d3b-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:31.642688 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.642651 5136 generic.go:356] "Generic (PLEG): container finished" podID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" containerID="2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479" exitCode=0 Nov 02 20:15:31.642895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.642700 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" Nov 02 20:15:31.642895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.642747 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" event={"ID":"496d118d-3398-4f92-aa8c-d8cc19c77d3b","Type":"ContainerDied","Data":"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479"} Nov 02 20:15:31.642895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.642819 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p" event={"ID":"496d118d-3398-4f92-aa8c-d8cc19c77d3b","Type":"ContainerDied","Data":"4b4e8a385f485bf5824c80a97f0d973d07be43c2df4251c1ea101a7a2137cc74"} Nov 02 20:15:31.642895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.642849 5136 scope.go:122] "RemoveContainer" containerID="2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479" Nov 02 20:15:31.646416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.646389 5136 generic.go:356] "Generic (PLEG): container finished" podID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerID="ef374d182ac03a16f3042909d53561b29fdf10699445b76c15dcc209f68d185d" exitCode=0 Nov 02 20:15:31.646567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.646551 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerDied","Data":"ef374d182ac03a16f3042909d53561b29fdf10699445b76c15dcc209f68d185d"} Nov 02 20:15:31.665491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.664716 5136 scope.go:122] "RemoveContainer" containerID="2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479" Nov 02 20:15:31.665909 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:31.665881 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479\": container with ID starting with 2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479 not found: ID does not exist" containerID="2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479" Nov 02 20:15:31.666018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.665917 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479"} err="failed to get container status \"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479\": rpc error: code = NotFound desc = could not find container \"2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479\": container with ID starting with 2736c9b193f8eb52eeee7bbf0a8a9c5a16503aad694eb4bef8651cbc4c1ca479 not found: ID does not exist" Nov 02 20:15:31.666018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.665940 5136 scope.go:122] "RemoveContainer" containerID="eb6ca35d90f557548174c988dbcca9da5326465deba04d0f9a04b9fe8cf22f80" Nov 02 20:15:31.772001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.771977 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:15:31.800495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.800466 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:15:31.803267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.803241 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-kxn5p"] Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948377 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948454 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948539 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948585 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948608 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948671 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948746 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948780 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.948873 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\") pod \"2601ac1b-87cc-42ef-8089-f36a3111d77c\" (UID: \"2601ac1b-87cc-42ef-8089-f36a3111d77c\") " Nov 02 20:15:31.949717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.949516 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:31.950370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.950052 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:31.950370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.950203 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:31.967536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:31.967232 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:32.019666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.012779 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:32.053983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.053869 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.053983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.053906 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.053983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.053923 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.053983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.053938 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601ac1b-87cc-42ef-8089-f36a3111d77c-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.053983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.053953 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.061944 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.061887 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:32.062198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.062166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:32.067065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.066966 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "kube-api-access-mt42g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:32.067065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.067040 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token" pod "2601ac1b-87cc-42ef-8089-f36a3111d77c" (UID: "2601ac1b-87cc-42ef-8089-f36a3111d77c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:32.154895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.154863 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.154895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.154892 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.155086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.154904 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt42g\" (UniqueName: \"kubernetes.io/projected/2601ac1b-87cc-42ef-8089-f36a3111d77c-kube-api-access-mt42g\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.155086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.154914 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601ac1b-87cc-42ef-8089-f36a3111d77c-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:32.547353 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.547315 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" path="/var/lib/kubelet/pods/496d118d-3398-4f92-aa8c-d8cc19c77d3b/volumes" Nov 02 20:15:32.655806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.655769 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" event={"ID":"2601ac1b-87cc-42ef-8089-f36a3111d77c","Type":"ContainerDied","Data":"dde5ac0f17bbd47f005b7f501987241709aae671d19cc3bdf9e36df6887f4bf7"} Nov 02 20:15:32.655981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.655845 5136 scope.go:122] "RemoveContainer" containerID="ef374d182ac03a16f3042909d53561b29fdf10699445b76c15dcc209f68d185d" Nov 02 20:15:32.655981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.655785 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd" Nov 02 20:15:32.863391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.863294 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:15:32.867249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:32.867217 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-08eeaa-openshift-default-5bb5b8b769-r4dwd"] Nov 02 20:15:34.548288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:34.548256 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" path="/var/lib/kubelet/pods/2601ac1b-87cc-42ef-8089-f36a3111d77c/volumes" Nov 02 20:15:38.273170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:38.273137 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:15:38.273820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:38.273366 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" containerID="cri-o://a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82" gracePeriod=30 Nov 02 20:15:43.546168 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.546106 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:15:43.631149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631121 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631176 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631266 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631324 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631344 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631392 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631433 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631461 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631501 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\") pod \"027a52bb-63d6-49fc-bf68-eb2727f275b9\" (UID: \"027a52bb-63d6-49fc-bf68-eb2727f275b9\") " Nov 02 20:15:43.631682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631656 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.631937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.631912 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632034 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632125 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632478 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632598 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632647 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632669 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.633194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.632685 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027a52bb-63d6-49fc-bf68-eb2727f275b9-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.633636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.633551 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.633857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.633828 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:43.633969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.633867 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "kube-api-access-c4bjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:43.634440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.634415 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo" pod "027a52bb-63d6-49fc-bf68-eb2727f275b9" (UID: "027a52bb-63d6-49fc-bf68-eb2727f275b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:43.804040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.803826 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.804040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.803856 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.804040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.803871 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.804040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.803885 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027a52bb-63d6-49fc-bf68-eb2727f275b9-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.804040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.803898 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4bjw\" (UniqueName: \"kubernetes.io/projected/027a52bb-63d6-49fc-bf68-eb2727f275b9-kube-api-access-c4bjw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.813471 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.813425 5136 generic.go:356] "Generic (PLEG): container finished" podID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerID="a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82" exitCode=0 Nov 02 20:15:43.813652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.813604 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" event={"ID":"027a52bb-63d6-49fc-bf68-eb2727f275b9","Type":"ContainerDied","Data":"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82"} Nov 02 20:15:43.813764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.813658 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" event={"ID":"027a52bb-63d6-49fc-bf68-eb2727f275b9","Type":"ContainerDied","Data":"9d1036c06ef055e0b6b760c3bf9c398cdd19c56cbdcacc38908f71f9b2371c08"} Nov 02 20:15:43.813764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.813685 5136 scope.go:122] "RemoveContainer" containerID="a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82" Nov 02 20:15:43.813873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.813861 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" Nov 02 20:15:43.848364 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.848255 5136 scope.go:122] "RemoveContainer" containerID="a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82" Nov 02 20:15:43.865301 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:43.865210 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82\": container with ID starting with a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82 not found: ID does not exist" containerID="a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82" Nov 02 20:15:43.865301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.865258 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82"} err="failed to get container status \"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82\": rpc error: code = NotFound desc = could not find container \"a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82\": container with ID starting with a51de7aef32dd0cae693db33ba1434a2e60ff6133bcb94b0b0de5e6f934b0e82 not found: ID does not exist" Nov 02 20:15:43.995704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:43.995670 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:15:44.001576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:44.001553 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv"] Nov 02 20:15:44.544162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:44.541252 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:44.544162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:44.541342 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-ccnmv" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:44.554291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:15:44.554257 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" path="/var/lib/kubelet/pods/027a52bb-63d6-49fc-bf68-eb2727f275b9/volumes" Nov 02 20:15:51.010114 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:15:51.009924 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:16:16.018564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.018530 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:16:16.019157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.018751 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" podUID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" containerName="maas-api" containerID="cri-o://57cc8aa65f1c3597c2c241bbdf461a59d8ceefccc2f07aaacd3a0ae9b979e5e6" gracePeriod=30 Nov 02 20:16:16.212764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.210707 5136 generic.go:356] "Generic (PLEG): container finished" podID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" containerID="57cc8aa65f1c3597c2c241bbdf461a59d8ceefccc2f07aaacd3a0ae9b979e5e6" exitCode=0 Nov 02 20:16:16.212764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.210907 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" event={"ID":"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb","Type":"ContainerDied","Data":"57cc8aa65f1c3597c2c241bbdf461a59d8ceefccc2f07aaacd3a0ae9b979e5e6"} Nov 02 20:16:16.371173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.371152 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:16:16.624803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.589103 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.624803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.589200 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.624803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.589349 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.624803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.589396 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.681516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.641360 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:16.684570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.681884 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "kube-api-access-gpcpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:16.728057 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.728028 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.728470 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:16:16.728340 5136 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb/volumes/kubernetes.io~configmap/openshift-service-ca Nov 02 20:16:16.728470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.728436 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:16.728769 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:16:16.728681 5136 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb/volumes/kubernetes.io~projected/kube-api-access-gpcpc Nov 02 20:16:16.728769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.728747 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "kube-api-access-gpcpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:16.728941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.728926 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") pod \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\" (UID: \"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb\") " Nov 02 20:16:16.740165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.740126 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:16.742276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.742251 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcpc\" (UniqueName: \"kubernetes.io/projected/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-kube-api-access-gpcpc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:16.772150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.772000 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:16.772150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.772093 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls" pod "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" (UID: "f2bef18d-ab8d-48b5-af70-8d0d7bd965bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:16.843534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.843503 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:16.843534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:16.843530 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:17.249756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:17.249580 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" event={"ID":"f2bef18d-ab8d-48b5-af70-8d0d7bd965bb","Type":"ContainerDied","Data":"089211336d1520d3a08ac65e9d82db717768dd1f390f288d0fb1ed6f4a8b35b8"} Nov 02 20:16:17.249756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:17.249657 5136 scope.go:122] "RemoveContainer" containerID="57cc8aa65f1c3597c2c241bbdf461a59d8ceefccc2f07aaacd3a0ae9b979e5e6" Nov 02 20:16:17.250272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:17.249859 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn" Nov 02 20:16:17.420897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:17.420869 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:16:17.423331 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:17.423299 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-341fb2-75f688dc74-n4nhn"] Nov 02 20:16:18.547741 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:18.547698 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" path="/var/lib/kubelet/pods/f2bef18d-ab8d-48b5-af70-8d0d7bd965bb/volumes" Nov 02 20:16:20.445270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:20.445235 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:16:20.445695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:20.445460 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" containerID="cri-o://ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f" gracePeriod=30 Nov 02 20:16:21.003859 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:21.003819 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:16:23.408038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408006 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:16:23.408602 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408587 5136 cpu_manager.go:414] "containerMap: removing container" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" Nov 02 20:16:23.408645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408603 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" Nov 02 20:16:23.408645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408620 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" containerName="maas-api" Nov 02 20:16:23.408645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408625 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" containerName="maas-api" Nov 02 20:16:23.408645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408635 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" containerName="maas-api" Nov 02 20:16:23.408645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408640 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" containerName="maas-api" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408648 5136 cpu_manager.go:414] "containerMap: removing container" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408652 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408663 5136 cpu_manager.go:414] "containerMap: removing container" podUID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" containerName="maas-api" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408668 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" containerName="maas-api" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408681 5136 cpu_manager.go:414] "containerMap: removing container" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408686 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408692 5136 cpu_manager.go:414] "containerMap: removing container" podUID="822a4355-c346-4689-bfe2-21344dedf120" containerName="maas-api" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408697 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822a4355-c346-4689-bfe2-21344dedf120" containerName="maas-api" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408706 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408712 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408719 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408723 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408756 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" Nov 02 20:16:23.408828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408763 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408924 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="6194de7e-caca-4b06-a83a-6a6a492b6bca" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408980 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bef18d-ab8d-48b5-af70-8d0d7bd965bb" containerName="maas-api" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.408992 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6ae38b-f81e-40dd-8d62-97fd6ab77401" containerName="maas-api" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409008 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5feec8-d535-4a7e-a1ea-33999b4f9978" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409019 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="496d118d-3398-4f92-aa8c-d8cc19c77d3b" containerName="maas-api" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409033 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="027a52bb-63d6-49fc-bf68-eb2727f275b9" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409042 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="822a4355-c346-4689-bfe2-21344dedf120" containerName="maas-api" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409051 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409060 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2601ac1b-87cc-42ef-8089-f36a3111d77c" containerName="istio-proxy" Nov 02 20:16:23.409340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.409076 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f166cc2-1183-4da4-94c3-6fcb437fbade" containerName="istio-proxy" Nov 02 20:16:23.412844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.412819 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.460954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.460925 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9c4\" (UniqueName: \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.461552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.461018 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.461552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.461080 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.461552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.461172 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.464952 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.464916 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:16:23.507921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.507892 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5e04421b-serving-cert\"" Nov 02 20:16:23.562365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.562340 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.562533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.562440 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9c4\" (UniqueName: \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.562533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.562485 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.562533 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:23.562519 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-serving-cert: secret "maas-api-e2e-mt-5e04421b-serving-cert" not found Nov 02 20:16:23.562711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.562538 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.562711 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:23.562602 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls podName:b4d0afd5-4336-416e-bc31-a30889b66e86 nodeName:}" failed. No retries permitted until 2026-11-02 20:16:24.062580048 +0000 UTC m=+1667.669124541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls") pod "maas-api-e2e-mt-5e04421b-84659b6669-j68kk" (UID: "b4d0afd5-4336-416e-bc31-a30889b66e86") : secret "maas-api-e2e-mt-5e04421b-serving-cert" not found Nov 02 20:16:23.563382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.563357 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.570424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.570400 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:23.617028 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:23.616996 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9c4\" (UniqueName: \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:24.107699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.107663 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:24.110123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.110097 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") pod \"maas-api-e2e-mt-5e04421b-84659b6669-j68kk\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:24.207306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.207269 5136 patch_prober.go:29] interesting pod/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:24.207472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.207331 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:24.326030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.326000 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:24.691976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.691941 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:16:24.693191 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:16:24.693165 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d0afd5_4336_416e_bc31_a30889b66e86.slice/crio-ed6ede3fe4b143d1689da55b3ba61783006e4e05ce99b524d36cf5a0e6e89989 WatchSource:0}: Error finding container ed6ede3fe4b143d1689da55b3ba61783006e4e05ce99b524d36cf5a0e6e89989: Status 404 returned error can't find the container with id ed6ede3fe4b143d1689da55b3ba61783006e4e05ce99b524d36cf5a0e6e89989 Nov 02 20:16:24.694994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:24.694976 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:16:25.320331 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.320282 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" event={"ID":"b4d0afd5-4336-416e-bc31-a30889b66e86","Type":"ContainerStarted","Data":"ed6ede3fe4b143d1689da55b3ba61783006e4e05ce99b524d36cf5a0e6e89989"} Nov 02 20:16:25.776050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.776016 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:16:25.934105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934028 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934075 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934198 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934289 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934318 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:25.934514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934351 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934390 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934661 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:25.934799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934686 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934780 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.934914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.934834 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\") pod \"33efceeb-0fd9-43f5-8245-2672bfc65adc\" (UID: \"33efceeb-0fd9-43f5-8245-2672bfc65adc\") " Nov 02 20:16:25.935340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935313 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:25.935560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935534 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:25.935643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935565 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:25.935643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935585 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33efceeb-0fd9-43f5-8245-2672bfc65adc-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:25.935643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935599 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:25.935865 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.935777 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:25.937657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.937632 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:25.938091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.938067 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:25.938173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.938100 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:25.938173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:25.938140 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz" pod "33efceeb-0fd9-43f5-8245-2672bfc65adc" (UID: "33efceeb-0fd9-43f5-8245-2672bfc65adc"). InnerVolumeSpecName "kube-api-access-hj8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038100 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038137 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038153 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038174 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038188 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.039040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.038203 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8xz\" (UniqueName: \"kubernetes.io/projected/33efceeb-0fd9-43f5-8245-2672bfc65adc-kube-api-access-hj8xz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:26.328163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.328110 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" event={"ID":"b4d0afd5-4336-416e-bc31-a30889b66e86","Type":"ContainerStarted","Data":"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61"} Nov 02 20:16:26.328359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.328216 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:26.329920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.329893 5136 generic.go:356] "Generic (PLEG): container finished" podID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerID="ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f" exitCode=0 Nov 02 20:16:26.330033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.329932 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerDied","Data":"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f"} Nov 02 20:16:26.330033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.329947 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" Nov 02 20:16:26.330033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.329964 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz" event={"ID":"33efceeb-0fd9-43f5-8245-2672bfc65adc","Type":"ContainerDied","Data":"f8f172e558ef28fb8d6c4b79026261ddb35b25f3fe978635a066215e1413c968"} Nov 02 20:16:26.330033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.329986 5136 scope.go:122] "RemoveContainer" containerID="ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f" Nov 02 20:16:26.341555 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.341538 5136 scope.go:122] "RemoveContainer" containerID="c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20" Nov 02 20:16:26.350180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.350159 5136 scope.go:122] "RemoveContainer" containerID="ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f" Nov 02 20:16:26.350462 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:26.350438 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f\": container with ID starting with ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f not found: ID does not exist" containerID="ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f" Nov 02 20:16:26.350597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.350469 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f"} err="failed to get container status \"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f\": rpc error: code = NotFound desc = could not find container \"ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f\": container with ID starting with ffb2bfbae287505e93c9df10d96107f461c6a141bbe6917dccc7059ae20a137f not found: ID does not exist" Nov 02 20:16:26.350597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.350485 5136 scope.go:122] "RemoveContainer" containerID="c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20" Nov 02 20:16:26.350825 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:26.350778 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20\": container with ID starting with c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20 not found: ID does not exist" containerID="c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20" Nov 02 20:16:26.350825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.350804 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20"} err="failed to get container status \"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20\": rpc error: code = NotFound desc = could not find container \"c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20\": container with ID starting with c351bebf1c51f847397c522c75167708fa5c76eeaf3e3d45fb6463309ecc3a20 not found: ID does not exist" Nov 02 20:16:26.437899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.437849 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" podStartSLOduration=2.5052625490000002 podStartE2EDuration="3.437833798s" totalImagesPullingTime="932.571249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:26.436200418 +0000 UTC m=+1670.042744918" watchObservedRunningTime="2026-11-02 20:16:26.437833798 +0000 UTC m=+1670.044378296" Nov 02 20:16:26.576759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.573851 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:16:26.576759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:26.573918 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-341fb2-openshift-default-68bf7c558c-46rbz"] Nov 02 20:16:28.382069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.382037 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:16:28.382497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.382243 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" podUID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" containerName="maas-api" containerID="cri-o://7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b" gracePeriod=30 Nov 02 20:16:28.547908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.547871 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" path="/var/lib/kubelet/pods/33efceeb-0fd9-43f5-8245-2672bfc65adc/volumes" Nov 02 20:16:28.747205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.747185 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:16:28.898763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.896321 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\" (UniqueName: \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\") pod \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " Nov 02 20:16:28.898763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.896411 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\") pod \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " Nov 02 20:16:28.910766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.900887 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\") pod \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " Nov 02 20:16:28.910766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.901018 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") pod \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\" (UID: \"55ca8acc-6a45-458e-be14-cc112e7d9e9f\") " Nov 02 20:16:28.910766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.901518 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca" pod "55ca8acc-6a45-458e-be14-cc112e7d9e9f" (UID: "55ca8acc-6a45-458e-be14-cc112e7d9e9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:28.910766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.901821 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ca8acc-6a45-458e-be14-cc112e7d9e9f-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:28.911690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.911647 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls" pod "55ca8acc-6a45-458e-be14-cc112e7d9e9f" (UID: "55ca8acc-6a45-458e-be14-cc112e7d9e9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:28.911851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.911751 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs" pod "55ca8acc-6a45-458e-be14-cc112e7d9e9f" (UID: "55ca8acc-6a45-458e-be14-cc112e7d9e9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:28.912681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:28.912650 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh" pod "55ca8acc-6a45-458e-be14-cc112e7d9e9f" (UID: "55ca8acc-6a45-458e-be14-cc112e7d9e9f"). InnerVolumeSpecName "kube-api-access-kg2dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:29.019806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.018402 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2dh\" (UniqueName: \"kubernetes.io/projected/55ca8acc-6a45-458e-be14-cc112e7d9e9f-kube-api-access-kg2dh\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:29.019806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.018442 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:29.019806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.018457 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ca8acc-6a45-458e-be14-cc112e7d9e9f-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:29.350841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.350764 5136 generic.go:356] "Generic (PLEG): container finished" podID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" containerID="7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b" exitCode=0 Nov 02 20:16:29.350841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.350803 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" Nov 02 20:16:29.350841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.350810 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" event={"ID":"55ca8acc-6a45-458e-be14-cc112e7d9e9f","Type":"ContainerDied","Data":"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b"} Nov 02 20:16:29.351072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.350858 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w" event={"ID":"55ca8acc-6a45-458e-be14-cc112e7d9e9f","Type":"ContainerDied","Data":"090ebbb5324c5c8090bbfae102dfc14843a071ef2c19841580c8cfed08bb0396"} Nov 02 20:16:29.351072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.350880 5136 scope.go:122] "RemoveContainer" containerID="7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b" Nov 02 20:16:29.363067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.363046 5136 scope.go:122] "RemoveContainer" containerID="7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b" Nov 02 20:16:29.363360 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:29.363339 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b\": container with ID starting with 7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b not found: ID does not exist" containerID="7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b" Nov 02 20:16:29.363430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.363371 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b"} err="failed to get container status \"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b\": rpc error: code = NotFound desc = could not find container \"7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b\": container with ID starting with 7ed75f9b0b4525059b06b636d7383f00801c557174b27fc54cff966eb77b217b not found: ID does not exist" Nov 02 20:16:29.517160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.517124 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:16:29.562914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.562886 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3021d-5c795576c7-x4m7w"] Nov 02 20:16:29.858004 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.857967 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 20:16:29.858249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.858208 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" containerID="cri-o://eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967" gracePeriod=30 Nov 02 20:16:29.860926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.860904 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:16:29.861142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.861105 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" containerID="cri-o://c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1" gracePeriod=30 Nov 02 20:16:29.912042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.912002 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:16:29.912230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.912128 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:16:29.912383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.912357 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" containerID="cri-o://10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c" gracePeriod=30 Nov 02 20:16:29.912674 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:29.912651 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" containerID="cri-o://39d24f0f1f462e6a019e890eb029137a859adcc4ac0e489ba8e54c966c19bc66" gracePeriod=30 Nov 02 20:16:30.551831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:30.551792 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" path="/var/lib/kubelet/pods/55ca8acc-6a45-458e-be14-cc112e7d9e9f/volumes" Nov 02 20:16:32.343264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:32.343236 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:16:34.324178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.324143 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dptmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:34.324639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.324195 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:34.429398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.429364 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:34.429564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.429420 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:34.481306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.481273 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-s66gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:34.481451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.481342 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:34.645260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.645173 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:16:34.645446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.645424 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerName="maas-api" containerID="cri-o://ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7" gracePeriod=30 Nov 02 20:16:34.746694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:34.746652 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": dial tcp 10.133.0.84:8443: connect: connection refused" Nov 02 20:16:35.158355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.158285 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:16:35.210839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.210018 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\") pod \"f3563925-8b17-4e84-a9ee-0644dbdbde48\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " Nov 02 20:16:35.210839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.210054 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") pod \"f3563925-8b17-4e84-a9ee-0644dbdbde48\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " Nov 02 20:16:35.210839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.210081 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\" (UniqueName: \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\") pod \"f3563925-8b17-4e84-a9ee-0644dbdbde48\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " Nov 02 20:16:35.210839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.210217 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\") pod \"f3563925-8b17-4e84-a9ee-0644dbdbde48\" (UID: \"f3563925-8b17-4e84-a9ee-0644dbdbde48\") " Nov 02 20:16:35.212953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.211520 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca" pod "f3563925-8b17-4e84-a9ee-0644dbdbde48" (UID: "f3563925-8b17-4e84-a9ee-0644dbdbde48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.215125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.215085 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls" pod "f3563925-8b17-4e84-a9ee-0644dbdbde48" (UID: "f3563925-8b17-4e84-a9ee-0644dbdbde48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:35.215213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.215149 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs" pod "f3563925-8b17-4e84-a9ee-0644dbdbde48" (UID: "f3563925-8b17-4e84-a9ee-0644dbdbde48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:35.215356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.215333 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj" pod "f3563925-8b17-4e84-a9ee-0644dbdbde48" (UID: "f3563925-8b17-4e84-a9ee-0644dbdbde48"). InnerVolumeSpecName "kube-api-access-2lrnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.240769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.238036 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:16:35.298971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.298936 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 20:16:35.311102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311073 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311161 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311216 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311254 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311280 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311329 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311362 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.311452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311402 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311447 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311481 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311518 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311559 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311590 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311615 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.311711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311674 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.312037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311719 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.312037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311772 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\") pod \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\" (UID: \"7f8ffee0-608f-4c70-bbfc-6d1f34a53460\") " Nov 02 20:16:35.312037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311815 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.312037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311845 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.312037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.311878 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\") pod \"d93ed1f6-54b8-4c85-8700-082702818464\" (UID: \"d93ed1f6-54b8-4c85-8700-082702818464\") " Nov 02 20:16:35.312351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312326 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.312450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312422 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312610 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312661 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312680 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312696 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3563925-8b17-4e84-a9ee-0644dbdbde48-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312710 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312726 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrnj\" (UniqueName: \"kubernetes.io/projected/f3563925-8b17-4e84-a9ee-0644dbdbde48-kube-api-access-2lrnj\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312757 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93ed1f6-54b8-4c85-8700-082702818464-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312772 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312785 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3563925-8b17-4e84-a9ee-0644dbdbde48-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.312980 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.313194 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.313257 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.313330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.313308 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.314231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.314166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.315426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.315359 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.315426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.315387 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.315426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.315402 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "kube-api-access-d7jg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.315625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.315445 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "kube-api-access-f98f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.315849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.315829 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo" pod "7f8ffee0-608f-4c70-bbfc-6d1f34a53460" (UID: "7f8ffee0-608f-4c70-bbfc-6d1f34a53460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:35.316320 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.316302 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:35.316524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.316333 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy" pod "d93ed1f6-54b8-4c85-8700-082702818464" (UID: "d93ed1f6-54b8-4c85-8700-082702818464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.347341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.347320 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:16:35.400391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.400355 5136 generic.go:356] "Generic (PLEG): container finished" podID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerID="ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7" exitCode=0 Nov 02 20:16:35.400595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.400413 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" Nov 02 20:16:35.400595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.400433 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" event={"ID":"f3563925-8b17-4e84-a9ee-0644dbdbde48","Type":"ContainerDied","Data":"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7"} Nov 02 20:16:35.400595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.400495 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5" event={"ID":"f3563925-8b17-4e84-a9ee-0644dbdbde48","Type":"ContainerDied","Data":"a7cb06bbf191eff674558df7b37346fa85e20d16b2c23639eeb30d3abec4e190"} Nov 02 20:16:35.400595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.400550 5136 scope.go:122] "RemoveContainer" containerID="ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7" Nov 02 20:16:35.402481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.402453 5136 generic.go:356] "Generic (PLEG): container finished" podID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerID="39d24f0f1f462e6a019e890eb029137a859adcc4ac0e489ba8e54c966c19bc66" exitCode=0 Nov 02 20:16:35.402598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.402558 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" event={"ID":"c5f127d3-0a80-491d-acc1-48cf7e1cae8a","Type":"ContainerDied","Data":"39d24f0f1f462e6a019e890eb029137a859adcc4ac0e489ba8e54c966c19bc66"} Nov 02 20:16:35.404938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.404785 5136 generic.go:356] "Generic (PLEG): container finished" podID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerID="eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967" exitCode=0 Nov 02 20:16:35.404938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.404822 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" event={"ID":"7f8ffee0-608f-4c70-bbfc-6d1f34a53460","Type":"ContainerDied","Data":"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967"} Nov 02 20:16:35.404938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.404844 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" Nov 02 20:16:35.404938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.404883 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r" event={"ID":"7f8ffee0-608f-4c70-bbfc-6d1f34a53460","Type":"ContainerDied","Data":"fedc24105f934e76b325a21713e154a00e668f283ff834371fd1929891a60c8a"} Nov 02 20:16:35.406837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.406815 5136 generic.go:356] "Generic (PLEG): container finished" podID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerID="10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c" exitCode=0 Nov 02 20:16:35.406984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.406965 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" Nov 02 20:16:35.407195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.407093 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" event={"ID":"37bc9082-4e2a-4ca0-ba36-76bfe456605b","Type":"ContainerDied","Data":"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c"} Nov 02 20:16:35.407195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.407135 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz" event={"ID":"37bc9082-4e2a-4ca0-ba36-76bfe456605b","Type":"ContainerDied","Data":"25b0c3955bc15331977489adac7569137780e07e81c8c2f3db5581f0da19e55b"} Nov 02 20:16:35.408640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.408619 5136 generic.go:356] "Generic (PLEG): container finished" podID="d93ed1f6-54b8-4c85-8700-082702818464" containerID="c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1" exitCode=0 Nov 02 20:16:35.408763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.408666 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" event={"ID":"d93ed1f6-54b8-4c85-8700-082702818464","Type":"ContainerDied","Data":"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1"} Nov 02 20:16:35.408763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.408703 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" event={"ID":"d93ed1f6-54b8-4c85-8700-082702818464","Type":"ContainerDied","Data":"7dbda460b5b5eea407dbf58b538fde8fe658e045f05ff44a9e859316a7103fe5"} Nov 02 20:16:35.408888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.408781 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413377 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413442 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413514 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413552 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413588 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413621 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413673 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413800 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.413912 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\") pod \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\" (UID: \"37bc9082-4e2a-4ca0-ba36-76bfe456605b\") " Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414432 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414456 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414471 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414485 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414499 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414515 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jg4\" (UniqueName: \"kubernetes.io/projected/d93ed1f6-54b8-4c85-8700-082702818464-kube-api-access-d7jg4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414530 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f98f9\" (UniqueName: \"kubernetes.io/projected/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-kube-api-access-f98f9\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414544 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414557 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93ed1f6-54b8-4c85-8700-082702818464-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414571 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414584 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414596 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414610 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ffee0-608f-4c70-bbfc-6d1f34a53460-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.414869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.414624 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93ed1f6-54b8-4c85-8700-082702818464-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.416114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.415285 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.416114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.415748 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.416114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.415932 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.416646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.416424 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.417583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.417492 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.419498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.419477 5136 scope.go:122] "RemoveContainer" containerID="ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7" Nov 02 20:16:35.419921 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:35.419895 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7\": container with ID starting with ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7 not found: ID does not exist" containerID="ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7" Nov 02 20:16:35.420018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.419927 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7"} err="failed to get container status \"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7\": rpc error: code = NotFound desc = could not find container \"ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7\": container with ID starting with ae4d1ae0d8000bba5af2ed46cd85f80372ada17f4436202b36136dbe328e9fc7 not found: ID does not exist" Nov 02 20:16:35.420018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.419945 5136 scope.go:122] "RemoveContainer" containerID="eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967" Nov 02 20:16:35.429122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.429098 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:35.433566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.433540 5136 scope.go:122] "RemoveContainer" containerID="eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967" Nov 02 20:16:35.434007 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:35.433983 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967\": container with ID starting with eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967 not found: ID does not exist" containerID="eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967" Nov 02 20:16:35.434103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.434018 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967"} err="failed to get container status \"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967\": rpc error: code = NotFound desc = could not find container \"eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967\": container with ID starting with eb635710949892de043b7792cbf673790ccdd00bc73a2a47fcd41baac7d99967 not found: ID does not exist" Nov 02 20:16:35.434103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.434040 5136 scope.go:122] "RemoveContainer" containerID="10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c" Nov 02 20:16:35.443054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.443024 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.443166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.443060 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.443166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.443089 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5" pod "37bc9082-4e2a-4ca0-ba36-76bfe456605b" (UID: "37bc9082-4e2a-4ca0-ba36-76bfe456605b"). InnerVolumeSpecName "kube-api-access-cxfr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.444406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.444378 5136 scope.go:122] "RemoveContainer" containerID="10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c" Nov 02 20:16:35.444662 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:35.444646 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c\": container with ID starting with 10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c not found: ID does not exist" containerID="10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c" Nov 02 20:16:35.444715 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.444674 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c"} err="failed to get container status \"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c\": rpc error: code = NotFound desc = could not find container \"10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c\": container with ID starting with 10ed9614252a00ca5115be72e943f842f0814ffa2b6e63aa8dc66a00b6b3de4c not found: ID does not exist" Nov 02 20:16:35.444715 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.444689 5136 scope.go:122] "RemoveContainer" containerID="c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1" Nov 02 20:16:35.452967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.452948 5136 scope.go:122] "RemoveContainer" containerID="c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1" Nov 02 20:16:35.453232 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:35.453215 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1\": container with ID starting with c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1 not found: ID does not exist" containerID="c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1" Nov 02 20:16:35.453295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.453242 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1"} err="failed to get container status \"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1\": rpc error: code = NotFound desc = could not find container \"c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1\": container with ID starting with c298e28add786e1153947fbbff802ed70c2ea5743e4556f80222b53d5ef558e1 not found: ID does not exist" Nov 02 20:16:35.517131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517095 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517131 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517144 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517156 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517168 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517179 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517191 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bc9082-4e2a-4ca0-ba36-76bfe456605b-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517206 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfr5\" (UniqueName: \"kubernetes.io/projected/37bc9082-4e2a-4ca0-ba36-76bfe456605b-kube-api-access-cxfr5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.517344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.517219 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bc9082-4e2a-4ca0-ba36-76bfe456605b-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.644721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.644696 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:16:35.719132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719107 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719161 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719244 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719312 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719444 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719488 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719526 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719554 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719551 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.719594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719579 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\") pod \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\" (UID: \"c5f127d3-0a80-491d-acc1-48cf7e1cae8a\") " Nov 02 20:16:35.719864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719652 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.719970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.719892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.720288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.720224 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.720288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.720253 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.720288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.720270 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.720288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.720228 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.720288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.720277 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.721468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.721445 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "kube-api-access-7gr2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.721628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.721605 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:35.722040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.722018 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:35.722126 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.722089 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token" pod "c5f127d3-0a80-491d-acc1-48cf7e1cae8a" (UID: "c5f127d3-0a80-491d-acc1-48cf7e1cae8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.804756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.799793 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825465 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825503 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825514 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825526 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825538 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.825966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.825551 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gr2d\" (UniqueName: \"kubernetes.io/projected/c5f127d3-0a80-491d-acc1-48cf7e1cae8a-kube-api-access-7gr2d\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.842791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.837083 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-s66gc"] Nov 02 20:16:35.877243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.877205 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 20:16:35.881093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.881065 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t8n2r"] Nov 02 20:16:35.983285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.983204 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:16:35.987372 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:35.987342 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-671e7c-56b4cf7478-jrvl5"] Nov 02 20:16:36.096958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.096927 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:16:36.099464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.099436 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dptmz"] Nov 02 20:16:36.416267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.416181 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" Nov 02 20:16:36.416267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.416202 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g" event={"ID":"c5f127d3-0a80-491d-acc1-48cf7e1cae8a","Type":"ContainerDied","Data":"86e0a03ad1b9c17ec0eb78bd50f5a6285bd18e456e2b27ae6906649726768fdb"} Nov 02 20:16:36.416802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.416279 5136 scope.go:122] "RemoveContainer" containerID="39d24f0f1f462e6a019e890eb029137a859adcc4ac0e489ba8e54c966c19bc66" Nov 02 20:16:36.547441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.547412 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" path="/var/lib/kubelet/pods/37bc9082-4e2a-4ca0-ba36-76bfe456605b/volumes" Nov 02 20:16:36.547929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.547910 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" path="/var/lib/kubelet/pods/7f8ffee0-608f-4c70-bbfc-6d1f34a53460/volumes" Nov 02 20:16:36.548439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.548421 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93ed1f6-54b8-4c85-8700-082702818464" path="/var/lib/kubelet/pods/d93ed1f6-54b8-4c85-8700-082702818464/volumes" Nov 02 20:16:36.549130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.549115 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" path="/var/lib/kubelet/pods/f3563925-8b17-4e84-a9ee-0644dbdbde48/volumes" Nov 02 20:16:36.572666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.572645 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:16:36.576109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:36.576089 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-n8w4g"] Nov 02 20:16:38.546923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:38.546890 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" path="/var/lib/kubelet/pods/c5f127d3-0a80-491d-acc1-48cf7e1cae8a/volumes" Nov 02 20:16:40.628690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:40.628659 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:16:40.629060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:40.628898 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" containerID="cri-o://da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4" gracePeriod=30 Nov 02 20:16:44.077076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:44.077040 5136 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:44.077472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:44.077103 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:45.894683 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.894627 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:16:45.968506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968477 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968546 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968566 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968631 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968674 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968706 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968766 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968791 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968820 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968864 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\") pod \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\" (UID: \"ff8a815a-46f5-4d8d-aaef-3228369bbbef\") " Nov 02 20:16:45.968885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.968873 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:45.969189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969122 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:45.969280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969251 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:45.969454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969435 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:45.969602 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969583 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:45.969673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969603 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:45.969673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969613 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:45.969673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969624 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:45.969673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.969638 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:45.971016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.970993 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "kube-api-access-vg2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:45.971128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.971096 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:45.971381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.971355 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:45.971451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:45.971395 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token" pod "ff8a815a-46f5-4d8d-aaef-3228369bbbef" (UID: "ff8a815a-46f5-4d8d-aaef-3228369bbbef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:46.070997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.070957 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.070997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.070992 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.071262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.071007 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2sk\" (UniqueName: \"kubernetes.io/projected/ff8a815a-46f5-4d8d-aaef-3228369bbbef-kube-api-access-vg2sk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.071262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.071020 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8a815a-46f5-4d8d-aaef-3228369bbbef-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.482689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.482655 5136 generic.go:356] "Generic (PLEG): container finished" podID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerID="da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4" exitCode=0 Nov 02 20:16:46.482879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.482698 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerDied","Data":"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4"} Nov 02 20:16:46.482879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.482782 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" event={"ID":"ff8a815a-46f5-4d8d-aaef-3228369bbbef","Type":"ContainerDied","Data":"a82f938d70a3aa8a8216ebb26792ed81912d09fd9e2f420e520bed094c9112ef"} Nov 02 20:16:46.482879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.482794 5136 scope.go:122] "RemoveContainer" containerID="da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4" Nov 02 20:16:46.482879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.482707 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6" Nov 02 20:16:46.493538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.493519 5136 scope.go:122] "RemoveContainer" containerID="ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5" Nov 02 20:16:46.501891 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.501874 5136 scope.go:122] "RemoveContainer" containerID="da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4" Nov 02 20:16:46.502143 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:46.502124 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4\": container with ID starting with da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4 not found: ID does not exist" containerID="da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4" Nov 02 20:16:46.502199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.502154 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4"} err="failed to get container status \"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4\": rpc error: code = NotFound desc = could not find container \"da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4\": container with ID starting with da1790c42d9452b75e87d6b39ff2e984b7459d6504873e163ad877560a7223c4 not found: ID does not exist" Nov 02 20:16:46.502199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.502170 5136 scope.go:122] "RemoveContainer" containerID="ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5" Nov 02 20:16:46.502408 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:46.502376 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5\": container with ID starting with ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5 not found: ID does not exist" containerID="ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5" Nov 02 20:16:46.502444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.502412 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5"} err="failed to get container status \"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5\": rpc error: code = NotFound desc = could not find container \"ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5\": container with ID starting with ca7df8c787721ecf4a9b123f37eee8934d606c422057b5420a6d703c712963f5 not found: ID does not exist" Nov 02 20:16:46.639012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.638980 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:16:46.686343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:46.686312 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-674b6"] Nov 02 20:16:48.556426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:48.556389 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" path="/var/lib/kubelet/pods/ff8a815a-46f5-4d8d-aaef-3228369bbbef/volumes" Nov 02 20:16:51.004443 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:51.004408 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:16:52.243919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.243889 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:16:52.244452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244427 5136 cpu_manager.go:414] "containerMap: removing container" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" Nov 02 20:16:52.244452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244444 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244469 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244477 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244488 5136 cpu_manager.go:414] "containerMap: removing container" podUID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" containerName="maas-api" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244496 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" containerName="maas-api" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244511 5136 cpu_manager.go:414] "containerMap: removing container" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244518 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244536 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244542 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244552 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244559 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244582 5136 cpu_manager.go:414] "containerMap: removing container" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244588 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244605 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244611 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244624 5136 cpu_manager.go:414] "containerMap: removing container" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244631 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244643 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerName="maas-api" Nov 02 20:16:52.244665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244651 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerName="maas-api" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244809 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bc9082-4e2a-4ca0-ba36-76bfe456605b" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244826 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f127d3-0a80-491d-acc1-48cf7e1cae8a" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244839 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ca8acc-6a45-458e-be14-cc112e7d9e9f" containerName="maas-api" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244849 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244859 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244868 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a815a-46f5-4d8d-aaef-3228369bbbef" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244879 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="33efceeb-0fd9-43f5-8245-2672bfc65adc" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244889 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8ffee0-608f-4c70-bbfc-6d1f34a53460" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244897 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93ed1f6-54b8-4c85-8700-082702818464" containerName="istio-proxy" Nov 02 20:16:52.245596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.244904 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3563925-8b17-4e84-a9ee-0644dbdbde48" containerName="maas-api" Nov 02 20:16:52.250186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.250167 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.356775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.356714 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.356935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.356861 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hdk\" (UniqueName: \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.356935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.356902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.357179 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.357154 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.360654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.360636 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-40aecd-serving-cert\"" Nov 02 20:16:52.363106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.363089 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:16:52.458060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.458028 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.458212 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:52.458173 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-serving-cert: secret "maas-api-e2e-api-b-40aecd-serving-cert" not found Nov 02 20:16:52.458212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.458176 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.458334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.458224 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.458334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.458325 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hdk\" (UniqueName: \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.458437 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:52.458376 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls podName:34abd91e-2472-42ee-9901-e33c608cabf5 nodeName:}" failed. No retries permitted until 2026-11-02 20:16:52.958347892 +0000 UTC m=+1696.564892381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls") pod "maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" (UID: "34abd91e-2472-42ee-9901-e33c608cabf5") : secret "maas-api-e2e-api-b-40aecd-serving-cert" not found Nov 02 20:16:52.458856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.458830 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.460840 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.460820 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.558850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.550364 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hdk\" (UniqueName: \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.977052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.977018 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:52.979477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:52.979452 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") pod \"maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:53.157385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.157351 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:16:53.161817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.161790 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.162886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.162856 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:53.211965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.211915 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f83bc2-openshift-default-dockercfg-kctzr\"" Nov 02 20:16:53.263127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.263097 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:16:53.281544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281509 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281588 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281628 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281661 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281765 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281903 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.281958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.281943 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.282071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.282052 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvml\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.282116 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.282087 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.382944 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.382910 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.382955 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383003 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvml\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383026 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383050 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383092 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383128 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383162 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383200 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383390 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383412 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383637 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383545 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383677 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.383911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.383889 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.385932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.385914 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.386121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.386105 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.437294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.437266 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.437446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.437424 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvml\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\") pod \"e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.479836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.479809 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:53.563992 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:16:53.563964 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34abd91e_2472_42ee_9901_e33c608cabf5.slice/crio-ceb8d66077e2594a8a239c88b2a941e704e6aa64552a9c6392bc5e84cd420aaa WatchSource:0}: Error finding container ceb8d66077e2594a8a239c88b2a941e704e6aa64552a9c6392bc5e84cd420aaa: Status 404 returned error can't find the container with id ceb8d66077e2594a8a239c88b2a941e704e6aa64552a9c6392bc5e84cd420aaa Nov 02 20:16:53.575092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.568339 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:16:53.625771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.623872 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" event={"ID":"34abd91e-2472-42ee-9901-e33c608cabf5","Type":"ContainerStarted","Data":"ceb8d66077e2594a8a239c88b2a941e704e6aa64552a9c6392bc5e84cd420aaa"} Nov 02 20:16:53.851451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.851427 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:16:53.853228 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:16:53.853188 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c1f526_3290_4984_b3fb_7dcfc7b1d9ee.slice/crio-48b1bd7c3ee3d5197ec0469c238d229672c2fbf359d0af446fc34570205c3689 WatchSource:0}: Error finding container 48b1bd7c3ee3d5197ec0469c238d229672c2fbf359d0af446fc34570205c3689: Status 404 returned error can't find the container with id 48b1bd7c3ee3d5197ec0469c238d229672c2fbf359d0af446fc34570205c3689 Nov 02 20:16:53.856142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.856095 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:53.856221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.856190 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:53.856262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:53.856233 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:54.631194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.631163 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerStarted","Data":"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754"} Nov 02 20:16:54.631560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.631219 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerStarted","Data":"48b1bd7c3ee3d5197ec0469c238d229672c2fbf359d0af446fc34570205c3689"} Nov 02 20:16:54.632802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.632781 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" event={"ID":"34abd91e-2472-42ee-9901-e33c608cabf5","Type":"ContainerStarted","Data":"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c"} Nov 02 20:16:54.632902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.632823 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:16:54.742739 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.742678 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podStartSLOduration=1.7426652649999999 podStartE2EDuration="1.742665265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:54.740117504 +0000 UTC m=+1698.346662003" watchObservedRunningTime="2026-11-02 20:16:54.742665265 +0000 UTC m=+1698.349209763" Nov 02 20:16:54.849327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:54.849232 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" podStartSLOduration=1.888412167 podStartE2EDuration="2.849217892s" totalImagesPullingTime="960.805725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:54.847377058 +0000 UTC m=+1698.453921556" watchObservedRunningTime="2026-11-02 20:16:54.849217892 +0000 UTC m=+1698.455762384" Nov 02 20:16:55.480405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:55.480373 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:55.485998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:55.485974 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:55.638698 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:55.638667 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:55.639518 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:55.639501 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:16:56.876753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:56.876702 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:16:56.877231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:56.876935 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" podUID="c05920b8-5726-48d2-b219-bdd3b213a039" containerName="maas-api" containerID="cri-o://8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc" gracePeriod=30 Nov 02 20:16:57.211996 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.211972 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:16:57.265388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265356 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\" (UniqueName: \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\") pod \"c05920b8-5726-48d2-b219-bdd3b213a039\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " Nov 02 20:16:57.265552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265403 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\") pod \"c05920b8-5726-48d2-b219-bdd3b213a039\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " Nov 02 20:16:57.265552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265469 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") pod \"c05920b8-5726-48d2-b219-bdd3b213a039\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " Nov 02 20:16:57.265552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265511 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\") pod \"c05920b8-5726-48d2-b219-bdd3b213a039\" (UID: \"c05920b8-5726-48d2-b219-bdd3b213a039\") " Nov 02 20:16:57.265821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265797 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca" pod "c05920b8-5726-48d2-b219-bdd3b213a039" (UID: "c05920b8-5726-48d2-b219-bdd3b213a039"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:57.265938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.265925 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05920b8-5726-48d2-b219-bdd3b213a039-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:57.267528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.267506 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx" pod "c05920b8-5726-48d2-b219-bdd3b213a039" (UID: "c05920b8-5726-48d2-b219-bdd3b213a039"). InnerVolumeSpecName "kube-api-access-p5xgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:57.267620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.267550 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs" pod "c05920b8-5726-48d2-b219-bdd3b213a039" (UID: "c05920b8-5726-48d2-b219-bdd3b213a039"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:57.267620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.267570 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls" pod "c05920b8-5726-48d2-b219-bdd3b213a039" (UID: "c05920b8-5726-48d2-b219-bdd3b213a039"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:57.366618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.366589 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5xgx\" (UniqueName: \"kubernetes.io/projected/c05920b8-5726-48d2-b219-bdd3b213a039-kube-api-access-p5xgx\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:57.366618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.366617 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:57.366866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.366628 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05920b8-5726-48d2-b219-bdd3b213a039-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:57.650494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.650455 5136 generic.go:356] "Generic (PLEG): container finished" podID="c05920b8-5726-48d2-b219-bdd3b213a039" containerID="8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc" exitCode=0 Nov 02 20:16:57.650666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.650503 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" Nov 02 20:16:57.650666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.650540 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" event={"ID":"c05920b8-5726-48d2-b219-bdd3b213a039","Type":"ContainerDied","Data":"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc"} Nov 02 20:16:57.650666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.650601 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx" event={"ID":"c05920b8-5726-48d2-b219-bdd3b213a039","Type":"ContainerDied","Data":"4b68515ca6e4bb8c11e89526c42e93be65a63ec1959c32c7f9e98c9be13b60a5"} Nov 02 20:16:57.650666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.650624 5136 scope.go:122] "RemoveContainer" containerID="8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc" Nov 02 20:16:57.661334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.661317 5136 scope.go:122] "RemoveContainer" containerID="8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc" Nov 02 20:16:57.661596 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:16:57.661577 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc\": container with ID starting with 8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc not found: ID does not exist" containerID="8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc" Nov 02 20:16:57.661659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.661603 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc"} err="failed to get container status \"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc\": rpc error: code = NotFound desc = could not find container \"8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc\": container with ID starting with 8b07eb2f55c89be159b6f557bd9f547b324622837c3fd87d23c6ec756c923ecc not found: ID does not exist" Nov 02 20:16:57.807151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.807113 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:16:57.810143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:57.810120 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0aa5c-66ffdd78b5-br7jx"] Nov 02 20:16:58.587167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:58.587131 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05920b8-5726-48d2-b219-bdd3b213a039" path="/var/lib/kubelet/pods/c05920b8-5726-48d2-b219-bdd3b213a039/volumes" Nov 02 20:16:59.490855 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.490823 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:16:59.491438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.491414 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c05920b8-5726-48d2-b219-bdd3b213a039" containerName="maas-api" Nov 02 20:16:59.491438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.491430 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05920b8-5726-48d2-b219-bdd3b213a039" containerName="maas-api" Nov 02 20:16:59.491580 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.491545 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05920b8-5726-48d2-b219-bdd3b213a039" containerName="maas-api" Nov 02 20:16:59.496529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.496506 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.602212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.601670 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a999d7-openshift-default-dockercfg-st5k7\"" Nov 02 20:16:59.632610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.632577 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:16:59.634165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.633709 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.634501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.634480 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.634653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.634636 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.634926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.634905 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.635142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.635123 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.635261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.635245 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.635390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.635374 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.635500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.635485 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjgk\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.635606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.635594 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.736976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.736942 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737051 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737080 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737139 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737221 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737248 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737285 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737362 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737419 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjgk\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737473 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737550 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.737817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.737602 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.738412 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.738318 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.738548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.738478 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.740202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.740102 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.740377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.740359 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.791755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.791671 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.791755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.791712 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjgk\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\") pod \"e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:16:59.810512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:16:59.810488 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:17:00.238412 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.238388 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:17:00.239218 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:00.239190 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6fa59c_2b08_493e_8bf3_e6394f774236.slice/crio-44d1b312a4911a19037eaf5acb5cdbdd06a9381afbb55dcff0aca8b751e05410 WatchSource:0}: Error finding container 44d1b312a4911a19037eaf5acb5cdbdd06a9381afbb55dcff0aca8b751e05410: Status 404 returned error can't find the container with id 44d1b312a4911a19037eaf5acb5cdbdd06a9381afbb55dcff0aca8b751e05410 Nov 02 20:17:00.241617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.241584 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:00.241694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.241648 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:00.241694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.241678 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:00.644384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.644299 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:17:00.683614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.683575 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerStarted","Data":"b9e30b52b61bb92c12676b615867e1b18dd716f7b01fe2e52f01c213237f99d5"} Nov 02 20:17:00.683837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.683644 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerStarted","Data":"44d1b312a4911a19037eaf5acb5cdbdd06a9381afbb55dcff0aca8b751e05410"} Nov 02 20:17:00.811680 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.811649 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:17:00.864202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:00.864143 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podStartSLOduration=1.864115796 podStartE2EDuration="1.864115796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:00.862594255 +0000 UTC m=+1704.469138754" watchObservedRunningTime="2026-11-02 20:17:00.864115796 +0000 UTC m=+1704.470660296" Nov 02 20:17:01.817170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:01.817129 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:17:02.695440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:02.695398 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:17:02.696452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:02.696432 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:17:11.714819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.714788 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:17:11.718632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.718611 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.826704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.826676 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-296ae9-openshift-default-dockercfg-m2ptd\"" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.860022 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.903987 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.904051 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.904080 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.904112 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.904186 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.905591 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7fqk\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.908846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.905660 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.952398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.905721 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:11.952398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:11.912150 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:17:12.012708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012614 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.012708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012676 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.012708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012700 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012747 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012814 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012846 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7fqk\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012894 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.012947 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013253 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.013022 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013253 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.013194 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.013296 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.013430 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.013684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.013659 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.014138 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.014119 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.015788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.015762 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.015888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.015863 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.069874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.069844 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.070005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.069925 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7fqk\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\") pod \"e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.335993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.335893 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:12.800708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.800683 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:17:12.801552 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:12.801507 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a352b3_8451_4503_9ef8_c599a21831eb.slice/crio-06efcba7fcb97ba0e808df9abbb851541d8b1b0ca0f9ab3a8e72f84a43d588c9 WatchSource:0}: Error finding container 06efcba7fcb97ba0e808df9abbb851541d8b1b0ca0f9ab3a8e72f84a43d588c9: Status 404 returned error can't find the container with id 06efcba7fcb97ba0e808df9abbb851541d8b1b0ca0f9ab3a8e72f84a43d588c9 Nov 02 20:17:12.804248 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.804215 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:12.804367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.804289 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:12.804367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:12.804339 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:13.794823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:13.794784 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerStarted","Data":"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef"} Nov 02 20:17:13.794975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:13.794844 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerStarted","Data":"06efcba7fcb97ba0e808df9abbb851541d8b1b0ca0f9ab3a8e72f84a43d588c9"} Nov 02 20:17:14.001393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:14.001343 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podStartSLOduration=3.001328124 podStartE2EDuration="3.001328124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:13.952429035 +0000 UTC m=+1717.558973533" watchObservedRunningTime="2026-11-02 20:17:14.001328124 +0000 UTC m=+1717.607872622" Nov 02 20:17:14.336890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:14.336860 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:14.342235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:14.342211 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:14.801644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:14.801613 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:14.802389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:14.802366 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:17:15.215587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.215553 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq"] Nov 02 20:17:15.219531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.219509 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.288785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.271753 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\"" Nov 02 20:17:15.298633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.298590 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq"] Nov 02 20:17:15.371947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.371914 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372048 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372117 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-kube-api-access-cdktn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372225 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372279 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372412 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372361 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17725dd-2440-4da1-93ec-41e643aaf992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372425 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17725dd-2440-4da1-93ec-41e643aaf992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372486 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.372565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.372518 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.473924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.473850 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.473924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.473897 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-kube-api-access-cdktn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.473930 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.473959 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.473987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17725dd-2440-4da1-93ec-41e643aaf992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474041 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17725dd-2440-4da1-93ec-41e643aaf992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474064 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474111 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474160 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474277 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474341 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474561 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474506 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474573 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.474854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.474832 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17725dd-2440-4da1-93ec-41e643aaf992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.476304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.476282 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17725dd-2440-4da1-93ec-41e643aaf992-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.476541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.476519 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17725dd-2440-4da1-93ec-41e643aaf992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.528021 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.527992 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.528172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.528100 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/f17725dd-2440-4da1-93ec-41e643aaf992-kube-api-access-cdktn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-ns7hq\" (UID: \"f17725dd-2440-4da1-93ec-41e643aaf992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.532991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.532969 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:15.957180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.957154 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq"] Nov 02 20:17:15.959504 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:15.959469 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17725dd_2440_4da1_93ec_41e643aaf992.slice/crio-b642c0b75b635b8c87e8beed86b1e11bcd8d32f51e532e7dd320a928180dcecc WatchSource:0}: Error finding container b642c0b75b635b8c87e8beed86b1e11bcd8d32f51e532e7dd320a928180dcecc: Status 404 returned error can't find the container with id b642c0b75b635b8c87e8beed86b1e11bcd8d32f51e532e7dd320a928180dcecc Nov 02 20:17:15.961999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.961967 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:15.962721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.962686 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:15.963190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:15.962947 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:16.821203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:16.821163 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" event={"ID":"f17725dd-2440-4da1-93ec-41e643aaf992","Type":"ContainerStarted","Data":"2fea8642d2adbea8e74531361889ddbe931844895499d64d3d551b1792c35180"} Nov 02 20:17:16.821725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:16.821230 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" event={"ID":"f17725dd-2440-4da1-93ec-41e643aaf992","Type":"ContainerStarted","Data":"b642c0b75b635b8c87e8beed86b1e11bcd8d32f51e532e7dd320a928180dcecc"} Nov 02 20:17:16.986590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:16.986542 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podStartSLOduration=1.986528121 podStartE2EDuration="1.986528121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:16.984221607 +0000 UTC m=+1720.590766104" watchObservedRunningTime="2026-11-02 20:17:16.986528121 +0000 UTC m=+1720.593072618" Nov 02 20:17:17.533855 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:17.533800 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:17.535713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:17.535685 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:17.535888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:17.535763 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:18.534025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:18.533986 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:18.534493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:18.534040 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:19.533956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:19.533919 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:19.534135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:19.533971 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:20.533777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:20.533714 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:20.533979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:20.533801 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:21.025805 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:21.025767 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:17:21.533463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:21.533429 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:21.533632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:21.533482 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:22.533404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:22.533370 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:22.533940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:22.533433 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:23.533796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:23.533756 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:17:23.534320 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:23.533810 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 20:17:24.593224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:24.593188 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:24.593678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:24.593578 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:24.594606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:24.594585 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" Nov 02 20:17:25.990185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:25.990144 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7522-d87633"] Nov 02 20:17:25.995776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:25.995743 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:26.042850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.042828 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-d87633"] Nov 02 20:17:26.086672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.086648 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\"" Nov 02 20:17:26.118677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.118646 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fn97\" (UniqueName: \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\") pod \"test-curl-7522-d87633\" (UID: \"53d97a56-9395-40d8-82b8-f1946cda8f36\") " pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:26.221185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.221147 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fn97\" (UniqueName: \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\") pod \"test-curl-7522-d87633\" (UID: \"53d97a56-9395-40d8-82b8-f1946cda8f36\") " pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:26.277426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.277352 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fn97\" (UniqueName: \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\") pod \"test-curl-7522-d87633\" (UID: \"53d97a56-9395-40d8-82b8-f1946cda8f36\") " pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:26.308801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.308726 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:26.691303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:26.691266 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-d87633"] Nov 02 20:17:26.692647 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:26.692601 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d97a56_9395_40d8_82b8_f1946cda8f36.slice/crio-4a9d95df25e9f1179954fa838c27e898d0ea785f8664d03e282583fd9d9df402 WatchSource:0}: Error finding container 4a9d95df25e9f1179954fa838c27e898d0ea785f8664d03e282583fd9d9df402: Status 404 returned error can't find the container with id 4a9d95df25e9f1179954fa838c27e898d0ea785f8664d03e282583fd9d9df402 Nov 02 20:17:27.004423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:27.004379 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-d87633" event={"ID":"53d97a56-9395-40d8-82b8-f1946cda8f36","Type":"ContainerStarted","Data":"4a9d95df25e9f1179954fa838c27e898d0ea785f8664d03e282583fd9d9df402"} Nov 02 20:17:29.016801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:29.016759 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-d87633" event={"ID":"53d97a56-9395-40d8-82b8-f1946cda8f36","Type":"ContainerStarted","Data":"2e2ee98de495529a31b44bafd8bf5af6f020441a2f177daf88eb741310e8bb98"} Nov 02 20:17:29.125411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:29.125355 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7522-d87633" podStartSLOduration=2.549433205 podStartE2EDuration="4.125337448s" totalImagesPullingTime="1.575904243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:29.122903978 +0000 UTC m=+1732.729448480" watchObservedRunningTime="2026-11-02 20:17:29.125337448 +0000 UTC m=+1732.731881947" Nov 02 20:17:30.333204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.333170 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:17:30.337703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.337683 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.380607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.380580 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:17:30.384550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.384534 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.430847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.430817 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:17:30.444769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.438517 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:17:30.475980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.475947 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.475992 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476068 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476094 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476133 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476234 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476257 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476288 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476494 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476554 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476640 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476675 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476711 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476827 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476850 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8jh\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.476874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476865 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdtw\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.476947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.476894 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.578497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578458 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.578667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578516 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8jh\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.578667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578548 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdtw\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.578667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578589 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.578667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578647 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578682 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578719 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578761 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578803 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578877 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578908 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578949 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579055 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.578978 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579088 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579137 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579144 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579179 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579181 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579212 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.579449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579245 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579565 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579802 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579878 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.579979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.579878 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.580088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.580066 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.580156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.580106 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.580156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.580137 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.580549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.580524 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.582203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.582173 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.582293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.582179 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.582717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.582684 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.582717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.582698 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.601052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.600987 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7522-d87633"] Nov 02 20:17:30.605629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.605603 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7522-d87633"] Nov 02 20:17:30.633925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.633891 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8jh\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.634067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.633891 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdtw\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.634108 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.634058 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7cjlt\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.635229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.635203 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cmhkb\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:30.652356 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.652329 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:30.697545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:30.697508 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:31.028670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.028605 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7522-d87633" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" containerName="test-curl-7522-d87633" containerID="cri-o://2e2ee98de495529a31b44bafd8bf5af6f020441a2f177daf88eb741310e8bb98" gracePeriod=2 Nov 02 20:17:31.067106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.067075 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:17:31.077293 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:31.077249 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-d87633\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:31.102843 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:31.102807 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6e9c49_fd36_4eb0_9a4c_05a78d6824d3.slice/crio-5a8f6f7c03f326005bf9eeaa0b1e38f92934066a797b1ede09e4a6f2fc6be7fd WatchSource:0}: Error finding container 5a8f6f7c03f326005bf9eeaa0b1e38f92934066a797b1ede09e4a6f2fc6be7fd: Status 404 returned error can't find the container with id 5a8f6f7c03f326005bf9eeaa0b1e38f92934066a797b1ede09e4a6f2fc6be7fd Nov 02 20:17:31.105411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.105378 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:31.105497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.105444 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:31.105497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.105473 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:31.116006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.115983 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:17:31.116878 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:31.116851 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108d74e8_b7d8_40ee_9f42_fde49744dc5f.slice/crio-c6067c266e09133a4727f7c46fd3875a60df2ff90f46a803a052b8ba36330c51 WatchSource:0}: Error finding container c6067c266e09133a4727f7c46fd3875a60df2ff90f46a803a052b8ba36330c51: Status 404 returned error can't find the container with id c6067c266e09133a4727f7c46fd3875a60df2ff90f46a803a052b8ba36330c51 Nov 02 20:17:31.119620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.119587 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:31.119689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.119651 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:31.119689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:31.119681 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:32.056463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.056420 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" event={"ID":"108d74e8-b7d8-40ee-9f42-fde49744dc5f","Type":"ContainerStarted","Data":"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f"} Nov 02 20:17:32.057118 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.057082 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" event={"ID":"108d74e8-b7d8-40ee-9f42-fde49744dc5f","Type":"ContainerStarted","Data":"c6067c266e09133a4727f7c46fd3875a60df2ff90f46a803a052b8ba36330c51"} Nov 02 20:17:32.058238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.058212 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" event={"ID":"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3","Type":"ContainerStarted","Data":"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75"} Nov 02 20:17:32.058338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.058265 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" event={"ID":"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3","Type":"ContainerStarted","Data":"5a8f6f7c03f326005bf9eeaa0b1e38f92934066a797b1ede09e4a6f2fc6be7fd"} Nov 02 20:17:32.108754 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:32.108107 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-d87633\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:32.330409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.330289 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podStartSLOduration=2.330269703 podStartE2EDuration="2.330269703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:32.329204903 +0000 UTC m=+1735.935749404" watchObservedRunningTime="2026-11-02 20:17:32.330269703 +0000 UTC m=+1735.936814202" Nov 02 20:17:32.443949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.443900 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podStartSLOduration=2.443884911 podStartE2EDuration="2.443884911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:32.442659334 +0000 UTC m=+1736.049203844" watchObservedRunningTime="2026-11-02 20:17:32.443884911 +0000 UTC m=+1736.050429409" Nov 02 20:17:32.491979 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:32.491934 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-d87633\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:32.652937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.652854 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:32.654020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.653995 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:32.654139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.654056 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:32.661340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.661303 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:17:32.662087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.662068 5136 cpu_manager.go:414] "containerMap: removing container" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" containerName="test-curl-7522-d87633" Nov 02 20:17:32.662183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.662088 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" containerName="test-curl-7522-d87633" Nov 02 20:17:32.662430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.662250 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" containerName="test-curl-7522-d87633" Nov 02 20:17:32.666082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.666063 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.702944 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.700282 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:32.721755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.707182 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:32.721755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.707235 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:32.741539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.722085 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e04421b-openshift-default-dockercfg-b5655\"" Nov 02 20:17:32.747779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.747611 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:17:32.845230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.845183 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.845516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.845491 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.845686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.845671 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.845905 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.845886 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gxv\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.846065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.846050 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.846199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.846186 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.846322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.846309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.846423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.846408 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.846569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.846553 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.947646 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.947844 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.947911 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gxv\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.947969 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948015 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948059 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948086 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948153 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948252 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.948843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.948718 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.949485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.949213 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.950781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.949898 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.950781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.950063 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.951382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.951358 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.953202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.953177 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:32.953575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:32.953532 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:33.027753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.023384 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:33.049556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.049522 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gxv\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:33.068084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.068052 5136 generic.go:356] "Generic (PLEG): container finished" podID="53d97a56-9395-40d8-82b8-f1946cda8f36" containerID="2e2ee98de495529a31b44bafd8bf5af6f020441a2f177daf88eb741310e8bb98" exitCode=137 Nov 02 20:17:33.280990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.280960 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:33.417755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.417044 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:33.566888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.566810 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\" (UniqueName: \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\") pod \"53d97a56-9395-40d8-82b8-f1946cda8f36\" (UID: \"53d97a56-9395-40d8-82b8-f1946cda8f36\") " Nov 02 20:17:33.569506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.569474 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97" pod "53d97a56-9395-40d8-82b8-f1946cda8f36" (UID: "53d97a56-9395-40d8-82b8-f1946cda8f36"). InnerVolumeSpecName "kube-api-access-9fn97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:33.652965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.652914 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:33.653193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.652973 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:33.669315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.669065 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fn97\" (UniqueName: \"kubernetes.io/projected/53d97a56-9395-40d8-82b8-f1946cda8f36-kube-api-access-9fn97\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:33.712208 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.712170 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:33.712504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.712475 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:33.822507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.822421 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:17:33.843869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.843831 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:33.843969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.843913 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:33.843969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:33.843941 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:34.077232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.077141 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" event={"ID":"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42","Type":"ContainerStarted","Data":"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88"} Nov 02 20:17:34.077232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.077208 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" event={"ID":"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42","Type":"ContainerStarted","Data":"239da9f48af223a2b4d67a6cfb8e5d579d8db034b7395ee0a8813124576bb2da"} Nov 02 20:17:34.079284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.079262 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:34.079410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.079320 5136 scope.go:122] "RemoveContainer" containerID="2e2ee98de495529a31b44bafd8bf5af6f020441a2f177daf88eb741310e8bb98" Nov 02 20:17:34.204756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.202957 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podStartSLOduration=2.202934287 podStartE2EDuration="2.202934287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:34.201647669 +0000 UTC m=+1737.808192167" watchObservedRunningTime="2026-11-02 20:17:34.202934287 +0000 UTC m=+1737.809478786" Nov 02 20:17:34.250608 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:34.250565 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-d87633\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" pod="opendatahub/test-curl-7522-d87633" Nov 02 20:17:34.282082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.282056 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:34.283262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.283238 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:34.283327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.283289 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:34.567782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.566793 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d97a56-9395-40d8-82b8-f1946cda8f36" path="/var/lib/kubelet/pods/53d97a56-9395-40d8-82b8-f1946cda8f36/volumes" Nov 02 20:17:34.653754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.653296 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:34.653754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.653354 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:34.705202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.705161 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:34.706433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:34.706400 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:35.281882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.281850 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:35.282332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.282306 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:35.662431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.657756 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:35.662431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.657827 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:35.702219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.702186 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:35.702502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:35.702473 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:36.281432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.281390 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:36.281608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.281447 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:36.667324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.666184 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:36.667324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.666253 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:36.704771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.701022 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:36.704771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:36.701078 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:37.281951 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.281917 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:37.282129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.281964 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:37.653338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.653295 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:37.653590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.653567 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:37.677260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.677226 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:17:37.683483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.683455 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.701817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.699402 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:37.701817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.699457 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:37.733879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.733851 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-40aecd-openshift-default-dockercfg-sm8fz\"" Nov 02 20:17:37.773032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.772996 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773151 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773205 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773267 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgpn\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773311 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773339 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773404 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773580 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773435 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.773580 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.773537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.879678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879643 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.879898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879715 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.879898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879796 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgpn\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.879898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879846 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.879898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879880 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.880150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.879948 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.880942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.880922 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.881243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.881227 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.882021 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.881863 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.882754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.882097 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.882754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.882169 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.882754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.882269 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.883066 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.883014 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.886301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.886278 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.894013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.893968 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.933058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.909406 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:17:37.943020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.942989 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.944225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.944164 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgpn\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:37.945704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:37.945659 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:38.025974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.025756 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:38.281630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.281595 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:38.281843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.281660 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:38.463024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.462999 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:17:38.465661 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:38.465626 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd542ec_e3da_42db_8ba6_0b3ba7a4421f.slice/crio-32739d7aa21768177e1f6a0710cbd273cd5ce0938f83c0667b121a5340f7b7b2 WatchSource:0}: Error finding container 32739d7aa21768177e1f6a0710cbd273cd5ce0938f83c0667b121a5340f7b7b2: Status 404 returned error can't find the container with id 32739d7aa21768177e1f6a0710cbd273cd5ce0938f83c0667b121a5340f7b7b2 Nov 02 20:17:38.469003 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.468966 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:38.469109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.469043 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:38.469109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.469087 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:38.653391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.653350 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:38.653559 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.653404 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:38.703966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.703928 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:38.704549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:38.704508 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:39.281638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.281594 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:39.281856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.281653 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:39.286509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.286478 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" event={"ID":"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f","Type":"ContainerStarted","Data":"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7"} Nov 02 20:17:39.286625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.286539 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" event={"ID":"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f","Type":"ContainerStarted","Data":"32739d7aa21768177e1f6a0710cbd273cd5ce0938f83c0667b121a5340f7b7b2"} Nov 02 20:17:39.654811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.654135 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:17:39.654811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.654200 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:17:39.710755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.710365 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:17:39.710755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:39.710422 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:17:40.029756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:40.028853 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:40.033775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:40.030850 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:17:40.033775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:40.030925 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:17:40.282376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:40.282289 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:17:40.282376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:40.282356 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 20:17:41.030191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.029575 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:17:41.030191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.029636 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:17:41.049769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.049720 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:41.050276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.050252 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:41.051565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.051533 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:17:41.070338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.070170 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:41.070662 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.070610 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:41.071626 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.071606 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:17:41.208445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.208385 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podStartSLOduration=4.208365477 podStartE2EDuration="4.208365477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:39.408344569 +0000 UTC m=+1743.014889069" watchObservedRunningTime="2026-11-02 20:17:41.208365477 +0000 UTC m=+1744.814909977" Nov 02 20:17:41.286492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.286425 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:41.286858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.286830 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:41.287503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:41.287486 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:17:42.026775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:42.026707 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:17:42.026933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:42.026798 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:17:43.026137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:43.026105 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:17:43.026668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:43.026165 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:17:44.026652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:44.026615 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:17:44.027067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:44.026687 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:17:45.030694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:45.030665 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:45.031142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:45.031128 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:45.031838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:45.031817 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:17:46.699854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.699812 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:17:46.707504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.707479 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.751293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.751262 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:17:46.795027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.795001 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f83bc2-serving-cert\"" Nov 02 20:17:46.807160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.807129 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.807275 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.807169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.807315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.807288 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjnh\" (UniqueName: \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.807432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.807413 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.908240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.908209 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.908438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.908337 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.908438 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:46.908347 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-serving-cert: secret "maas-api-e2e-ipp-a-f83bc2-serving-cert" not found Nov 02 20:17:46.908438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.908365 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.908438 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:46.908401 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls podName:96fb560a-8ea2-4852-b725-a2f9ce6c13ce nodeName:}" failed. No retries permitted until 2026-11-02 20:17:47.408386651 +0000 UTC m=+1751.014931127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls") pod "maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" (UID: "96fb560a-8ea2-4852-b725-a2f9ce6c13ce") : secret "maas-api-e2e-ipp-a-f83bc2-serving-cert" not found Nov 02 20:17:46.908664 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.908480 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjnh\" (UniqueName: \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.908974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.908955 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:46.910809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.910789 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:47.002890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:46.988118 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjnh\" (UniqueName: \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:47.420617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.420536 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:47.423015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.422987 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:47.622569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.622533 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:47.795814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.795784 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:17:47.801209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.801190 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.896678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.893055 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:17:47.926593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.926521 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.926593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.926566 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.926833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.926676 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.926899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.926876 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.926970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.926942 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224x6\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.927063 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.927047 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.927110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.927074 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.927110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.927092 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.927199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.927125 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:47.996134 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:47.996081 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:17:47.999660 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:47.999625 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fb560a_8ea2_4852_b725_a2f9ce6c13ce.slice/crio-c008ef2dcdde2cdb494450d5e2d4f1e8713a670f142609c945545292b55cfefe WatchSource:0}: Error finding container c008ef2dcdde2cdb494450d5e2d4f1e8713a670f142609c945545292b55cfefe: Status 404 returned error can't find the container with id c008ef2dcdde2cdb494450d5e2d4f1e8713a670f142609c945545292b55cfefe Nov 02 20:17:48.028343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028317 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028353 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224x6\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028390 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028432 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028468 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028659 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028828 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028866 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028896 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.028919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028907 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.029059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.028960 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.029151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.029132 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.029229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.029137 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.029465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.029440 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.030975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.030952 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.031456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.031435 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.083216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.083190 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.083371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.083356 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-224x6\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.115489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.115464 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:48.387095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.387062 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" event={"ID":"96fb560a-8ea2-4852-b725-a2f9ce6c13ce","Type":"ContainerStarted","Data":"c008ef2dcdde2cdb494450d5e2d4f1e8713a670f142609c945545292b55cfefe"} Nov 02 20:17:48.507752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.504953 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:17:48.523362 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:48.523325 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9795d192_2268_4b8a_8691_c27f8b7e3e22.slice/crio-1224f3eea9e7ce2b0b05c6ae645691eb5771c1a5896efb2ae8866f9b8beebad9 WatchSource:0}: Error finding container 1224f3eea9e7ce2b0b05c6ae645691eb5771c1a5896efb2ae8866f9b8beebad9: Status 404 returned error can't find the container with id 1224f3eea9e7ce2b0b05c6ae645691eb5771c1a5896efb2ae8866f9b8beebad9 Nov 02 20:17:48.526379 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.526338 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:48.526455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.526438 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:48.526501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:48.526486 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:49.416133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:49.416083 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" event={"ID":"96fb560a-8ea2-4852-b725-a2f9ce6c13ce","Type":"ContainerStarted","Data":"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992"} Nov 02 20:17:49.418135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:49.418108 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" event={"ID":"9795d192-2268-4b8a-8691-c27f8b7e3e22","Type":"ContainerStarted","Data":"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847"} Nov 02 20:17:49.418282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:49.418148 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" event={"ID":"9795d192-2268-4b8a-8691-c27f8b7e3e22","Type":"ContainerStarted","Data":"1224f3eea9e7ce2b0b05c6ae645691eb5771c1a5896efb2ae8866f9b8beebad9"} Nov 02 20:17:49.525217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:49.525164 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" podStartSLOduration=2.671293037 podStartE2EDuration="3.525151195s" totalImagesPullingTime="853.858158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:49.524653032 +0000 UTC m=+1753.131197530" watchObservedRunningTime="2026-11-02 20:17:49.525151195 +0000 UTC m=+1753.131695693" Nov 02 20:17:49.636547 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:49.636496 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podStartSLOduration=2.63648346 podStartE2EDuration="2.63648346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:49.635979436 +0000 UTC m=+1753.242523936" watchObservedRunningTime="2026-11-02 20:17:49.63648346 +0000 UTC m=+1753.243027959" Nov 02 20:17:50.120805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:50.116906 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:50.121206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:50.121174 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:50.121371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:50.121344 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:50.424711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:50.424627 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:51.024943 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:51.024885 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:17:51.116233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:51.116192 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:51.116411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:51.116262 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:52.115933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:52.115901 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:52.116352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:52.115960 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:53.116182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:53.116143 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:53.116900 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:53.116195 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:54.116522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:54.116486 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:54.117044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:54.116547 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:55.116554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:55.116520 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:17:55.116974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:55.116584 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:17:56.120717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:56.120690 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:56.121169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:56.121150 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:56.121838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:56.121822 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:17:56.437453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:56.437382 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:17:57.543702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.543673 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:17:57.547978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.547953 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.605339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.605303 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:17:57.613228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.613190 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.613383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.613262 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.613383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.613291 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrz2\" (UniqueName: \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.613489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.613388 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.644982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.644944 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a999d7-serving-cert\"" Nov 02 20:17:57.715185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.715142 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.715322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.715215 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.715322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.715243 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrz2\" (UniqueName: \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.715322 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:57.715280 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-serving-cert: secret "maas-api-e2e-ipp-a-a999d7-serving-cert" not found Nov 02 20:17:57.715477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.715347 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.715477 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:17:57.715439 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls podName:519b73a7-30e2-4c4b-bd95-22d336541988 nodeName:}" failed. No retries permitted until 2026-11-02 20:17:58.21541393 +0000 UTC m=+1761.821958413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls") pod "maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" (UID: "519b73a7-30e2-4c4b-bd95-22d336541988") : secret "maas-api-e2e-ipp-a-a999d7-serving-cert" not found Nov 02 20:17:57.716097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.716072 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.717794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.717767 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:57.771720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:57.771678 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrz2\" (UniqueName: \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:58.240968 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:58.240925 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:58.262466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:58.262434 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:58.462737 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:58.462694 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:17:58.836953 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:17:58.836925 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519b73a7_30e2_4c4b_bd95_22d336541988.slice/crio-5ca062e7dec62661029ad00137315a1905c1bfa1519154f8510fb19eeabac231 WatchSource:0}: Error finding container 5ca062e7dec62661029ad00137315a1905c1bfa1519154f8510fb19eeabac231: Status 404 returned error can't find the container with id 5ca062e7dec62661029ad00137315a1905c1bfa1519154f8510fb19eeabac231 Nov 02 20:17:58.837425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:58.837219 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:17:59.482611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:17:59.482573 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" event={"ID":"519b73a7-30e2-4c4b-bd95-22d336541988","Type":"ContainerStarted","Data":"5ca062e7dec62661029ad00137315a1905c1bfa1519154f8510fb19eeabac231"} Nov 02 20:18:00.490447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:00.490417 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" event={"ID":"519b73a7-30e2-4c4b-bd95-22d336541988","Type":"ContainerStarted","Data":"ad46f60823eb26ea09a193c92681f2e85dd3b4b5f96fa0d8b0ff15516ade22ae"} Nov 02 20:18:00.490897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:00.490523 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:18:00.600773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:00.600701 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" podStartSLOduration=2.73325481 podStartE2EDuration="3.600687834s" totalImagesPullingTime="867.433024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:00.598460893 +0000 UTC m=+1764.205005390" watchObservedRunningTime="2026-11-02 20:18:00.600687834 +0000 UTC m=+1764.207232331" Nov 02 20:18:02.913446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:02.913414 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:02.919014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:02.918991 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.008198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.008167 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:03.017331 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017291 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017510 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017363 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017510 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017424 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017510 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017482 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017546 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017611 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftmk\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017648 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017714 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.017911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.017760 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119626 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119581 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119643 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119694 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftmk\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119720 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119821 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.119853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119853 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.119973 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120023 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120046 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120086 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120111 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120287 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120414 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120368 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.120758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.120702 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.122323 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.122299 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.122405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.122393 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.176268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.176191 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.176483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.176456 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftmk\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.233320 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.233291 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:03.652275 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.652251 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:03.653004 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:18:03.652978 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8179a90_8c12_4f8b_8ac5_cf010f4bfbb5.slice/crio-885fc1f2a150b76806017b865605148aff21ec37fa6215ff0a63176924f974ad WatchSource:0}: Error finding container 885fc1f2a150b76806017b865605148aff21ec37fa6215ff0a63176924f974ad: Status 404 returned error can't find the container with id 885fc1f2a150b76806017b865605148aff21ec37fa6215ff0a63176924f974ad Nov 02 20:18:03.655785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.655753 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:03.655853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.655815 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:03.655853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:03.655846 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:04.570365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:04.570331 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" event={"ID":"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5","Type":"ContainerStarted","Data":"0850fa5cd48ce75748aaebbc264999aa90b7d4d251cf3d62d5883db46d3991f1"} Nov 02 20:18:04.570961 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:04.570377 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" event={"ID":"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5","Type":"ContainerStarted","Data":"885fc1f2a150b76806017b865605148aff21ec37fa6215ff0a63176924f974ad"} Nov 02 20:18:04.777466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:04.777402 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podStartSLOduration=2.777382396 podStartE2EDuration="2.777382396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:04.728198358 +0000 UTC m=+1768.334742855" watchObservedRunningTime="2026-11-02 20:18:04.777382396 +0000 UTC m=+1768.383926896" Nov 02 20:18:05.234004 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:05.233971 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:05.235687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:05.235659 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:18:05.235833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:05.235708 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 20:18:06.234507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:06.234470 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:18:06.234946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:06.234534 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 20:18:06.513841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:06.513753 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:18:07.234362 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:07.234329 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:18:07.234531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:07.234392 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 20:18:08.234182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:08.234149 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:18:08.234441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:08.234207 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 20:18:09.234226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:09.234182 5136 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:18:09.234633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:09.234257 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 20:18:10.238438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:10.238405 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:10.238838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:10.238825 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:10.239527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:10.239508 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:13.921375 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.921338 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:18:13.927162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.927140 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:13.956206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.956178 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:18:13.969348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.969306 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:13.969475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.969385 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:13.969552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.969522 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:13.970019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.969992 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2bz\" (UniqueName: \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:13.999966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:13.999926 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-296ae9-serving-cert\"" Nov 02 20:18:14.071831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.071803 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.071972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.071867 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.071972 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:14.071949 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-serving-cert: secret "maas-api-e2e-ipp-a-296ae9-serving-cert" not found Nov 02 20:18:14.072051 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:14.072030 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls podName:e274626b-2cba-47f3-8de6-dcf5d49b9870 nodeName:}" failed. No retries permitted until 2026-11-02 20:18:14.572014167 +0000 UTC m=+1778.178558643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls") pod "maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" (UID: "e274626b-2cba-47f3-8de6-dcf5d49b9870") : secret "maas-api-e2e-ipp-a-296ae9-serving-cert" not found Nov 02 20:18:14.072108 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.072094 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2bz\" (UniqueName: \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.072155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.072146 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.072801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.072776 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.074287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.074266 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.141032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.141006 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2bz\" (UniqueName: \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.596839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.596502 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.648972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.647279 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") pod \"maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:14.845045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:14.845004 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:15.619085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:15.619049 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:18:15.626496 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:18:15.626466 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode274626b_2cba_47f3_8de6_dcf5d49b9870.slice/crio-c0258dc0eea1334155c497ecc32750b00021486afad48326d27761c772037b2a WatchSource:0}: Error finding container c0258dc0eea1334155c497ecc32750b00021486afad48326d27761c772037b2a: Status 404 returned error can't find the container with id c0258dc0eea1334155c497ecc32750b00021486afad48326d27761c772037b2a Nov 02 20:18:15.875465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:15.875373 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" event={"ID":"e274626b-2cba-47f3-8de6-dcf5d49b9870","Type":"ContainerStarted","Data":"c0258dc0eea1334155c497ecc32750b00021486afad48326d27761c772037b2a"} Nov 02 20:18:16.884276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:16.884195 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" event={"ID":"e274626b-2cba-47f3-8de6-dcf5d49b9870","Type":"ContainerStarted","Data":"df3f6f46ec6241a73a710b79eeea6de9851bf5e4257fc8d6c75036381f67bcaf"} Nov 02 20:18:16.884276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:16.884254 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:17.016449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:17.016389 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" podStartSLOduration=3.10219006 podStartE2EDuration="4.016371179s" totalImagesPullingTime="914.181119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:17.000792215 +0000 UTC m=+1780.607336750" watchObservedRunningTime="2026-11-02 20:18:17.016371179 +0000 UTC m=+1780.622915677" Nov 02 20:18:18.853245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:18.853204 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:18:18.853766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:18.853467 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" podUID="b4d0afd5-4336-416e-bc31-a30889b66e86" containerName="maas-api" containerID="cri-o://fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61" gracePeriod=30 Nov 02 20:18:19.217851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.217825 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:18:19.295963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.295925 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") pod \"b4d0afd5-4336-416e-bc31-a30889b66e86\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " Nov 02 20:18:19.296176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.296017 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\") pod \"b4d0afd5-4336-416e-bc31-a30889b66e86\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " Nov 02 20:18:19.296253 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.296232 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\" (UniqueName: \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\") pod \"b4d0afd5-4336-416e-bc31-a30889b66e86\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " Nov 02 20:18:19.296321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.296301 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\") pod \"b4d0afd5-4336-416e-bc31-a30889b66e86\" (UID: \"b4d0afd5-4336-416e-bc31-a30889b66e86\") " Nov 02 20:18:19.296779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.296753 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca" pod "b4d0afd5-4336-416e-bc31-a30889b66e86" (UID: "b4d0afd5-4336-416e-bc31-a30889b66e86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:19.297033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.297008 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d0afd5-4336-416e-bc31-a30889b66e86-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:19.298333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.298311 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs" pod "b4d0afd5-4336-416e-bc31-a30889b66e86" (UID: "b4d0afd5-4336-416e-bc31-a30889b66e86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:19.298514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.298496 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls" pod "b4d0afd5-4336-416e-bc31-a30889b66e86" (UID: "b4d0afd5-4336-416e-bc31-a30889b66e86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:19.298604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.298589 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4" pod "b4d0afd5-4336-416e-bc31-a30889b66e86" (UID: "b4d0afd5-4336-416e-bc31-a30889b66e86"). InnerVolumeSpecName "kube-api-access-mm9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:19.404280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.402034 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:19.404280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.402074 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d0afd5-4336-416e-bc31-a30889b66e86-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:19.404280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.402094 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9c4\" (UniqueName: \"kubernetes.io/projected/b4d0afd5-4336-416e-bc31-a30889b66e86-kube-api-access-mm9c4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:19.920177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.920144 5136 generic.go:356] "Generic (PLEG): container finished" podID="b4d0afd5-4336-416e-bc31-a30889b66e86" containerID="fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61" exitCode=0 Nov 02 20:18:19.920642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.920200 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" Nov 02 20:18:19.920642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.920219 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" event={"ID":"b4d0afd5-4336-416e-bc31-a30889b66e86","Type":"ContainerDied","Data":"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61"} Nov 02 20:18:19.920642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.920272 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk" event={"ID":"b4d0afd5-4336-416e-bc31-a30889b66e86","Type":"ContainerDied","Data":"ed6ede3fe4b143d1689da55b3ba61783006e4e05ce99b524d36cf5a0e6e89989"} Nov 02 20:18:19.920642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.920296 5136 scope.go:122] "RemoveContainer" containerID="fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61" Nov 02 20:18:19.933316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.933300 5136 scope.go:122] "RemoveContainer" containerID="fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61" Nov 02 20:18:19.933584 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:19.933566 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61\": container with ID starting with fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61 not found: ID does not exist" containerID="fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61" Nov 02 20:18:19.933653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:19.933590 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61"} err="failed to get container status \"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61\": rpc error: code = NotFound desc = could not find container \"fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61\": container with ID starting with fdcf81f9f8bbc87f22519815a489a52fc5182e379cad832bece071064025cc61 not found: ID does not exist" Nov 02 20:18:20.118756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:20.116934 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:18:20.157020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:20.156987 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e04421b-84659b6669-j68kk"] Nov 02 20:18:20.548022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:20.547992 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d0afd5-4336-416e-bc31-a30889b66e86" path="/var/lib/kubelet/pods/b4d0afd5-4336-416e-bc31-a30889b66e86/volumes" Nov 02 20:18:21.005169 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:21.005134 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:18:21.937806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.937771 5136 generic.go:356] "Generic (PLEG): container finished" podID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerID="7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754" exitCode=0 Nov 02 20:18:21.938008 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.937852 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerDied","Data":"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754"} Nov 02 20:18:21.938289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.938272 5136 scope.go:122] "RemoveContainer" containerID="7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754" Nov 02 20:18:21.939817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.939791 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:21.939881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.939855 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:21.939920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:21.939888 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:22.916390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:22.916351 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:18:22.947127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:22.947097 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerStarted","Data":"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c"} Nov 02 20:18:23.024144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.024109 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:18:23.024784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.024769 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b4d0afd5-4336-416e-bc31-a30889b66e86" containerName="maas-api" Nov 02 20:18:23.024784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.024784 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d0afd5-4336-416e-bc31-a30889b66e86" containerName="maas-api" Nov 02 20:18:23.024886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.024876 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d0afd5-4336-416e-bc31-a30889b66e86" containerName="maas-api" Nov 02 20:18:23.027566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.027545 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.160286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.160153 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225097 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225154 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225212 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225259 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96v4g\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225306 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225365 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225395 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225533 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.231805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.225574 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.326504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326465 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.326702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326517 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.326702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326585 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.326702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326630 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96v4g\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.326702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326681 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326930 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.326974 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327009 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327157 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327205 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327605 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327649 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327769 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.327842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.327797 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.329909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.329883 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.330048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.330032 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.380869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.380838 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.381017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.380956 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96v4g\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:23.480363 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.480335 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:18:23.481691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.481669 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:23.481764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.481718 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:23.641429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:23.641398 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:24.078469 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:18:24.078432 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d6a818_2448_4dd1_b74e_97d1d73b3591.slice/crio-096ced5bb405b2689825afca759ee9202f5743c8b14feda7a9bc7dd17e00752b WatchSource:0}: Error finding container 096ced5bb405b2689825afca759ee9202f5743c8b14feda7a9bc7dd17e00752b: Status 404 returned error can't find the container with id 096ced5bb405b2689825afca759ee9202f5743c8b14feda7a9bc7dd17e00752b Nov 02 20:18:24.078885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.078499 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:18:24.081001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.080967 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:24.081083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.081032 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:24.081083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.081062 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:24.481163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.481129 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:24.481337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.481186 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:24.962813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.962777 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" event={"ID":"a1d6a818-2448-4dd1-b74e-97d1d73b3591","Type":"ContainerStarted","Data":"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153"} Nov 02 20:18:24.962988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:24.962845 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" event={"ID":"a1d6a818-2448-4dd1-b74e-97d1d73b3591","Type":"ContainerStarted","Data":"096ced5bb405b2689825afca759ee9202f5743c8b14feda7a9bc7dd17e00752b"} Nov 02 20:18:25.175231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.175165 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podStartSLOduration=3.175151668 podStartE2EDuration="3.175151668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:25.12730661 +0000 UTC m=+1788.733851109" watchObservedRunningTime="2026-11-02 20:18:25.175151668 +0000 UTC m=+1788.781696165" Nov 02 20:18:25.481108 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.481074 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:25.481273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.481121 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:25.550835 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.550801 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:18:25.551036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.551017 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" podUID="f73340a0-5315-462e-b5fa-ce602d9894ea" containerName="maas-api" containerID="cri-o://a26272eaf53c91469cf0d01488d153e8979a02cdaec1af60b3186f4d005d5c04" gracePeriod=30 Nov 02 20:18:25.643873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.643835 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:18:25.652179 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.652043 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:25.652924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.652898 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:25.653094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:25.653072 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:26.467010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.466831 5136 generic.go:356] "Generic (PLEG): container finished" podID="f73340a0-5315-462e-b5fa-ce602d9894ea" containerID="a26272eaf53c91469cf0d01488d153e8979a02cdaec1af60b3186f4d005d5c04" exitCode=0 Nov 02 20:18:26.467010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.466925 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" event={"ID":"f73340a0-5315-462e-b5fa-ce602d9894ea","Type":"ContainerDied","Data":"a26272eaf53c91469cf0d01488d153e8979a02cdaec1af60b3186f4d005d5c04"} Nov 02 20:18:26.480249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.480225 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:18:26.480757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.480715 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:26.480851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.480798 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:26.544966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.544928 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\") pod \"f73340a0-5315-462e-b5fa-ce602d9894ea\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " Nov 02 20:18:26.545150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.544996 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\" (UniqueName: \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\") pod \"f73340a0-5315-462e-b5fa-ce602d9894ea\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " Nov 02 20:18:26.545150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.545092 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") pod \"f73340a0-5315-462e-b5fa-ce602d9894ea\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " Nov 02 20:18:26.545150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.545144 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\") pod \"f73340a0-5315-462e-b5fa-ce602d9894ea\" (UID: \"f73340a0-5315-462e-b5fa-ce602d9894ea\") " Nov 02 20:18:26.546938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.546892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca" pod "f73340a0-5315-462e-b5fa-ce602d9894ea" (UID: "f73340a0-5315-462e-b5fa-ce602d9894ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:26.551871 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.551842 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92" pod "f73340a0-5315-462e-b5fa-ce602d9894ea" (UID: "f73340a0-5315-462e-b5fa-ce602d9894ea"). InnerVolumeSpecName "kube-api-access-kbf92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:26.551995 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.551901 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs" pod "f73340a0-5315-462e-b5fa-ce602d9894ea" (UID: "f73340a0-5315-462e-b5fa-ce602d9894ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:26.552252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.552233 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls" pod "f73340a0-5315-462e-b5fa-ce602d9894ea" (UID: "f73340a0-5315-462e-b5fa-ce602d9894ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:26.650419 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.650337 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:26.650657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.650629 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:26.659319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.658804 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:26.659483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.659460 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73340a0-5315-462e-b5fa-ce602d9894ea-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:26.659556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.659484 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73340a0-5315-462e-b5fa-ce602d9894ea-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:26.659556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:26.659499 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbf92\" (UniqueName: \"kubernetes.io/projected/f73340a0-5315-462e-b5fa-ce602d9894ea-kube-api-access-kbf92\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:27.469643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.469562 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:27.470342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.470307 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" containerID="cri-o://0850fa5cd48ce75748aaebbc264999aa90b7d4d251cf3d62d5883db46d3991f1" gracePeriod=30 Nov 02 20:18:27.482824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.480576 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:27.482824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.480641 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:27.555261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.553925 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:18:27.555261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.554216 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" containerID="cri-o://ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88" gracePeriod=30 Nov 02 20:18:27.562071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.562024 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:18:27.562381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.562343 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" containerID="cri-o://576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847" gracePeriod=30 Nov 02 20:18:27.565746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.565411 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" Nov 02 20:18:27.569655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.566120 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c" event={"ID":"f73340a0-5315-462e-b5fa-ce602d9894ea","Type":"ContainerDied","Data":"717b4c81e497acf1f183e8a3a7a116581ac12bfe7c0be5b1b0b7bbe3f1c1fd98"} Nov 02 20:18:27.569655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.566176 5136 scope.go:122] "RemoveContainer" containerID="a26272eaf53c91469cf0d01488d153e8979a02cdaec1af60b3186f4d005d5c04" Nov 02 20:18:27.668646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.665902 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:27.668646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:27.665968 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:28.046763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.046713 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:18:28.051198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.051170 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-15203f-5994dfd6d8-w4t9c"] Nov 02 20:18:28.480832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.480802 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 20:18:28.481237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.480848 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 20:18:28.548879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.548848 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73340a0-5315-462e-b5fa-ce602d9894ea" path="/var/lib/kubelet/pods/f73340a0-5315-462e-b5fa-ce602d9894ea/volumes" Nov 02 20:18:28.642045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.642009 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:28.642222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:28.642062 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:29.484704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:29.484645 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:18:29.485964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:29.485930 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:18:29.642800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:29.642765 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:29.643056 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:29.642828 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:30.642099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:30.642064 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:18:30.642533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:30.642141 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:18:31.647043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:31.647013 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:31.647494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:31.647306 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:31.648080 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:31.648059 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:18:32.642325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.642300 5136 generic.go:356] "Generic (PLEG): container finished" podID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerID="0850fa5cd48ce75748aaebbc264999aa90b7d4d251cf3d62d5883db46d3991f1" exitCode=0 Nov 02 20:18:32.642448 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.642371 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" event={"ID":"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5","Type":"ContainerDied","Data":"0850fa5cd48ce75748aaebbc264999aa90b7d4d251cf3d62d5883db46d3991f1"} Nov 02 20:18:32.828668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.828638 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:32.897801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.895496 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:18:32.915250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.915228 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:18:32.931812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.931787 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.931957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.931828 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.931991 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932097 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932180 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932258 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.932350 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932280 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932350 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932311 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932355 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932413 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\") pod \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\" (UID: \"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5\") " Nov 02 20:18:32.932578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932519 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.932931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932906 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.933022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.932989 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:32.933153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.933131 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.933222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.933161 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.933222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.933178 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.933222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.933194 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.933222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.933192 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.934469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.934433 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:32.935191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.935165 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.935289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.935189 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "kube-api-access-xftmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.936465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:32.936444 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy" pod "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" (UID: "b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.034110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034023 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034084 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034120 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034170 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034215 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034258 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034302 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034329 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034392 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034435 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034476 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034521 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034598 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034627 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034748 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034777 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\") pod \"9795d192-2268-4b8a-8691-c27f8b7e3e22\" (UID: \"9795d192-2268-4b8a-8691-c27f8b7e3e22\") " Nov 02 20:18:33.034980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034881 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.034980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.034925 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\") pod \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\" (UID: \"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42\") " Nov 02 20:18:33.035644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.035616 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftmk\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-kube-api-access-xftmk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.035781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.035646 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.035781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.035663 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.035781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.035677 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.035781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.035692 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.036488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.036462 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.037529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.037186 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.037740 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.037553 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:33.037740 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.037585 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.037941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.037917 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.038302 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.038276 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.038398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.038358 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:33.038671 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.038644 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.038794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.038682 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.038921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.038897 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.040359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.040334 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.040451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.040373 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "kube-api-access-97gxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.040613 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.040586 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.040978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.040860 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "kube-api-access-224x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.040978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.040956 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.041393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.041371 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:33.041772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.041742 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo" pod "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" (UID: "6a6b947b-9c39-42f9-a4d0-f15dca1b1d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:33.041906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.041889 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token" pod "9795d192-2268-4b8a-8691-c27f8b7e3e22" (UID: "9795d192-2268-4b8a-8691-c27f8b7e3e22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.137963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.137929 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.137963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.137965 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.137981 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.137999 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138016 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138032 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138048 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138063 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138078 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97gxv\" (UniqueName: \"kubernetes.io/projected/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-kube-api-access-97gxv\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138093 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138107 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138121 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-224x6\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-kube-api-access-224x6\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138136 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9795d192-2268-4b8a-8691-c27f8b7e3e22-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138150 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138163 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9795d192-2268-4b8a-8691-c27f8b7e3e22-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138177 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9795d192-2268-4b8a-8691-c27f8b7e3e22-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138192 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.138221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.138207 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.677679 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.677640 5136 generic.go:356] "Generic (PLEG): container finished" podID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerID="576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847" exitCode=0 Nov 02 20:18:33.678104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.678075 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" event={"ID":"9795d192-2268-4b8a-8691-c27f8b7e3e22","Type":"ContainerDied","Data":"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847"} Nov 02 20:18:33.678292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.678276 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" event={"ID":"9795d192-2268-4b8a-8691-c27f8b7e3e22","Type":"ContainerDied","Data":"1224f3eea9e7ce2b0b05c6ae645691eb5771c1a5896efb2ae8866f9b8beebad9"} Nov 02 20:18:33.678403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.678391 5136 scope.go:122] "RemoveContainer" containerID="576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847" Nov 02 20:18:33.680392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.679807 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6" Nov 02 20:18:33.682963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.682934 5136 generic.go:356] "Generic (PLEG): container finished" podID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerID="ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88" exitCode=0 Nov 02 20:18:33.683096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.683021 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" event={"ID":"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42","Type":"ContainerDied","Data":"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88"} Nov 02 20:18:33.683096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.683068 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" event={"ID":"6a6b947b-9c39-42f9-a4d0-f15dca1b1d42","Type":"ContainerDied","Data":"239da9f48af223a2b4d67a6cfb8e5d579d8db034b7395ee0a8813124576bb2da"} Nov 02 20:18:33.683220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.683155 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc" Nov 02 20:18:33.685253 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.685224 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" event={"ID":"b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5","Type":"ContainerDied","Data":"885fc1f2a150b76806017b865605148aff21ec37fa6215ff0a63176924f974ad"} Nov 02 20:18:33.685367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.685287 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx" Nov 02 20:18:33.697150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.697131 5136 scope.go:122] "RemoveContainer" containerID="576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847" Nov 02 20:18:33.697437 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:33.697410 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847\": container with ID starting with 576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847 not found: ID does not exist" containerID="576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847" Nov 02 20:18:33.697514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.697446 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847"} err="failed to get container status \"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847\": rpc error: code = NotFound desc = could not find container \"576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847\": container with ID starting with 576f63c99e86fc59cf852b9d112f98ce7fadfdad2f7c4275dd095fc915576847 not found: ID does not exist" Nov 02 20:18:33.697514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.697463 5136 scope.go:122] "RemoveContainer" containerID="ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88" Nov 02 20:18:33.707598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.707575 5136 scope.go:122] "RemoveContainer" containerID="ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88" Nov 02 20:18:33.707905 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:33.707880 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88\": container with ID starting with ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88 not found: ID does not exist" containerID="ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88" Nov 02 20:18:33.707998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.707910 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88"} err="failed to get container status \"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88\": rpc error: code = NotFound desc = could not find container \"ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88\": container with ID starting with ade0771ab7d954be97c176a37b649651cd6e515c00f6a4caeffeea9aba684c88 not found: ID does not exist" Nov 02 20:18:33.707998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.707933 5136 scope.go:122] "RemoveContainer" containerID="0850fa5cd48ce75748aaebbc264999aa90b7d4d251cf3d62d5883db46d3991f1" Nov 02 20:18:33.837250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.837164 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:18:33.884605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.884576 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-bggmc"] Nov 02 20:18:33.987645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.987606 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:33.991650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:33.991625 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-rh4lx"] Nov 02 20:18:34.095627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:34.095558 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:18:34.102500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:34.102474 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-mcdt6"] Nov 02 20:18:34.552712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:34.552676 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" path="/var/lib/kubelet/pods/6a6b947b-9c39-42f9-a4d0-f15dca1b1d42/volumes" Nov 02 20:18:34.553481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:34.553449 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" path="/var/lib/kubelet/pods/9795d192-2268-4b8a-8691-c27f8b7e3e22/volumes" Nov 02 20:18:34.553956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:34.553941 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" path="/var/lib/kubelet/pods/b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5/volumes" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.158229 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159232 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f73340a0-5315-462e-b5fa-ce602d9894ea" containerName="maas-api" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159257 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73340a0-5315-462e-b5fa-ce602d9894ea" containerName="maas-api" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159338 5136 cpu_manager.go:414] "containerMap: removing container" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159347 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159363 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159371 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159384 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159391 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159542 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6b947b-9c39-42f9-a4d0-f15dca1b1d42" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159554 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9795d192-2268-4b8a-8691-c27f8b7e3e22" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159567 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8179a90-8c12-4f8b-8ac5-cf010f4bfbb5" containerName="istio-proxy" Nov 02 20:18:38.159765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.159579 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73340a0-5315-462e-b5fa-ce602d9894ea" containerName="maas-api" Nov 02 20:18:38.205751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.193323 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.282091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.275466 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:18:38.366967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.366934 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.366985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367079 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367120 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367262 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367313 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnh5v\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367363 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367407 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.367494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.367457 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.468530 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468500 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.468699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468602 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.468699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468638 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.468699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468663 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.468886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468861 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.468986 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469000 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469047 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469071 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469087 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnh5v\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469119 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469147 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469152 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.469295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.469169 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.471243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.471217 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.471565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.471546 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.550769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.550242 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnh5v\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.575768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.575581 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:38.868047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:38.867961 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:39.288046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.288019 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:18:39.289517 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:18:39.289489 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32754173_8f0a_4223_87d3_dbc285464b8d.slice/crio-9161466b0408e05110fa00ae988074966d0020623e3c25e4a83c7887e8051482 WatchSource:0}: Error finding container 9161466b0408e05110fa00ae988074966d0020623e3c25e4a83c7887e8051482: Status 404 returned error can't find the container with id 9161466b0408e05110fa00ae988074966d0020623e3c25e4a83c7887e8051482 Nov 02 20:18:39.291363 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.291323 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:39.291447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.291399 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:39.291447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.291427 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:39.816340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.816293 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" event={"ID":"32754173-8f0a-4223-87d3-dbc285464b8d","Type":"ContainerStarted","Data":"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b"} Nov 02 20:18:39.816616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.816594 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" event={"ID":"32754173-8f0a-4223-87d3-dbc285464b8d","Type":"ContainerStarted","Data":"9161466b0408e05110fa00ae988074966d0020623e3c25e4a83c7887e8051482"} Nov 02 20:18:39.868934 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.868897 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:39.978877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:39.978827 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podStartSLOduration=1.978812135 podStartE2EDuration="1.978812135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:39.975839714 +0000 UTC m=+1803.582384205" watchObservedRunningTime="2026-11-02 20:18:39.978812135 +0000 UTC m=+1803.585356610" Nov 02 20:18:40.869929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:40.869900 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:18:40.870380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:40.870354 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 20:18:41.869106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:41.869073 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:18:41.869384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:41.869356 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 20:18:42.872474 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:42.871475 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:18:42.872474 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:42.871548 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 20:18:43.895289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:43.882477 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:18:43.895289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:43.882548 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 20:18:44.869089 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:44.869059 5136 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:18:44.869249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:44.869104 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 20:18:46.032020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:46.031992 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:46.033018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:46.032868 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:46.033979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:46.033960 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:18:48.736919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:48.736874 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:18:48.737352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:48.736973 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:18:49.986696 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.986665 5136 generic.go:356] "Generic (PLEG): container finished" podID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerID="b9e30b52b61bb92c12676b615867e1b18dd716f7b01fe2e52f01c213237f99d5" exitCode=0 Nov 02 20:18:49.987140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.986747 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerDied","Data":"b9e30b52b61bb92c12676b615867e1b18dd716f7b01fe2e52f01c213237f99d5"} Nov 02 20:18:49.987190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.987147 5136 scope.go:122] "RemoveContainer" containerID="b9e30b52b61bb92c12676b615867e1b18dd716f7b01fe2e52f01c213237f99d5" Nov 02 20:18:49.988652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.988611 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:49.988793 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.988695 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:49.988793 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:49.988777 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:51.019011 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:51.018970 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerStarted","Data":"f785e58772ea1fd69ef72ccb1ba05f02ee9c545d486b085fc93483d7c76ebf0a"} Nov 02 20:18:51.023573 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:18:51.023542 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:18:51.811122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:51.811091 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:18:51.811459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:51.811438 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:18:51.811605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:51.811583 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:18:52.813099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:52.813065 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:18:52.813708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:52.813678 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:18:53.813296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:53.813260 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:18:53.813839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:53.813513 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:18:54.813305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:54.813084 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:18:54.813305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:54.813148 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:18:55.812282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:55.811092 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:18:55.812282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:55.811163 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:18:56.847107 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:56.847073 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:18:56.848426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:56.848405 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:18:56.849569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:18:56.849545 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:19:10.119618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:10.119580 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:19:10.196196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:10.196159 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" containerID="cri-o://68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3" gracePeriod=30 Nov 02 20:19:13.999587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:13.999551 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:19:14.000174 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:13.999835 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" containerID="cri-o://54c75d37c38d94a7a6ead835fe255fd0d26b4088e3ca85300a9743fa1beb286c" gracePeriod=30 Nov 02 20:19:15.477050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.477025 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:19:15.582813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.582784 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.582857 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.582897 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.582968 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583022 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583081 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583115 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583152 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583173 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.583368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583298 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\") pod \"659b69c2-464c-4ed8-a7c2-8178fc81a723\" (UID: \"659b69c2-464c-4ed8-a7c2-8178fc81a723\") " Nov 02 20:19:15.583570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583359 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.583623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583596 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.583795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583761 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:15.583941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583850 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.583941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583873 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.583941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583887 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.584129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.583953 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.585279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.585256 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:15.585427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.585403 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.585547 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.585528 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:15.585611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.585568 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs" pod "659b69c2-464c-4ed8-a7c2-8178fc81a723" (UID: "659b69c2-464c-4ed8-a7c2-8178fc81a723"). InnerVolumeSpecName "kube-api-access-8wrvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685714 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685759 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685774 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659b69c2-464c-4ed8-a7c2-8178fc81a723-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685789 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685804 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wrvs\" (UniqueName: \"kubernetes.io/projected/659b69c2-464c-4ed8-a7c2-8178fc81a723-kube-api-access-8wrvs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.685846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:15.685820 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659b69c2-464c-4ed8-a7c2-8178fc81a723-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:16.245283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.245245 5136 generic.go:356] "Generic (PLEG): container finished" podID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerID="68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3" exitCode=0 Nov 02 20:19:16.245461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.245299 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" Nov 02 20:19:16.245461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.245326 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" event={"ID":"659b69c2-464c-4ed8-a7c2-8178fc81a723","Type":"ContainerDied","Data":"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3"} Nov 02 20:19:16.245461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.245406 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl" event={"ID":"659b69c2-464c-4ed8-a7c2-8178fc81a723","Type":"ContainerDied","Data":"581aaf1c9de37f602a39758ff7cb385ea2dc09e10f753c38e87192ff18ce5fdc"} Nov 02 20:19:16.245461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.245430 5136 scope.go:122] "RemoveContainer" containerID="68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3" Nov 02 20:19:16.256495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.256473 5136 scope.go:122] "RemoveContainer" containerID="68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3" Nov 02 20:19:16.256793 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:16.256759 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3\": container with ID starting with 68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3 not found: ID does not exist" containerID="68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3" Nov 02 20:19:16.256914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.256795 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3"} err="failed to get container status \"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3\": rpc error: code = NotFound desc = could not find container \"68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3\": container with ID starting with 68b9f0f78e45e01d88d9ed9eb087ee665ac4bad02f53dad4f0abbebcc64561f3 not found: ID does not exist" Nov 02 20:19:16.468110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.468077 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:19:16.473086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.473060 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-xtmcl"] Nov 02 20:19:16.547492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:16.547416 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" path="/var/lib/kubelet/pods/659b69c2-464c-4ed8-a7c2-8178fc81a723/volumes" Nov 02 20:19:19.385670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.385643 5136 generic.go:356] "Generic (PLEG): container finished" podID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerID="54c75d37c38d94a7a6ead835fe255fd0d26b4088e3ca85300a9743fa1beb286c" exitCode=0 Nov 02 20:19:19.386014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.385693 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" event={"ID":"d54ea449-0c01-48ec-91b2-ad42b3830434","Type":"ContainerDied","Data":"54c75d37c38d94a7a6ead835fe255fd0d26b4088e3ca85300a9743fa1beb286c"} Nov 02 20:19:19.429569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.429545 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:19:19.500244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.500207 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.500521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.500497 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.504794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.504769 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505024 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505198 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505320 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505496 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505638 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.505781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.505765 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\") pod \"d54ea449-0c01-48ec-91b2-ad42b3830434\" (UID: \"d54ea449-0c01-48ec-91b2-ad42b3830434\") " Nov 02 20:19:19.514908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.510998 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:19.514908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.511131 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:19.514908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.511256 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:19.514908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.511940 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:19.515205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.515178 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:19.515845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.515788 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:19.516049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.515957 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:19.516153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516071 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:19.516274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516218 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46" pod "d54ea449-0c01-48ec-91b2-ad42b3830434" (UID: "d54ea449-0c01-48ec-91b2-ad42b3830434"). InnerVolumeSpecName "kube-api-access-58d46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:19.516274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516265 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.516408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516299 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.516408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516315 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.516408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516332 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.516408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516346 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.516408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.516361 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.633260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.633168 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54ea449-0c01-48ec-91b2-ad42b3830434-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.633260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.633216 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54ea449-0c01-48ec-91b2-ad42b3830434-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:19.633260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:19.633240 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58d46\" (UniqueName: \"kubernetes.io/projected/d54ea449-0c01-48ec-91b2-ad42b3830434-kube-api-access-58d46\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:20.414514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:20.414483 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" Nov 02 20:19:20.415040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:20.414481 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c" event={"ID":"d54ea449-0c01-48ec-91b2-ad42b3830434","Type":"ContainerDied","Data":"38cb387138a8c2e67efc1ec70e3600a239805ebe0b0a5470634032946d6d1c02"} Nov 02 20:19:20.415040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:20.414680 5136 scope.go:122] "RemoveContainer" containerID="54c75d37c38d94a7a6ead835fe255fd0d26b4088e3ca85300a9743fa1beb286c" Nov 02 20:19:20.621270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:20.621235 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:19:20.668758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:20.668668 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-42q4c"] Nov 02 20:19:21.333788 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:21.333715 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:19:22.548787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:22.548750 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" path="/var/lib/kubelet/pods/d54ea449-0c01-48ec-91b2-ad42b3830434/volumes" Nov 02 20:19:26.459074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.459037 5136 generic.go:356] "Generic (PLEG): container finished" podID="28a352b3-8451-4503-9ef8-c599a21831eb" containerID="69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef" exitCode=0 Nov 02 20:19:26.459905 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.459157 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerDied","Data":"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef"} Nov 02 20:19:26.459905 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.459629 5136 scope.go:122] "RemoveContainer" containerID="69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef" Nov 02 20:19:26.461538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.461504 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:26.461650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.461592 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:26.461650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:26.461636 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:27.466912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:27.466880 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerStarted","Data":"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65"} Nov 02 20:19:28.336244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:28.336192 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:19:28.337546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:28.337507 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:19:28.337667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:28.337564 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:19:29.336454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:29.336419 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:19:29.336870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:29.336473 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:19:29.812985 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:29.811824 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:19:30.392752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:30.390214 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:19:30.392752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:30.390274 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:19:31.336885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:31.336852 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:19:31.337080 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:31.336913 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:19:32.336379 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:32.336342 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:19:32.336888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:32.336415 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:19:32.613540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:32.613466 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:19:32.613684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:32.613661 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" podUID="34abd91e-2472-42ee-9901-e33c608cabf5" containerName="maas-api" containerID="cri-o://665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c" gracePeriod=30 Nov 02 20:19:32.946463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:32.946441 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:19:33.047277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.047233 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") pod \"34abd91e-2472-42ee-9901-e33c608cabf5\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " Nov 02 20:19:33.047624 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.047598 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\") pod \"34abd91e-2472-42ee-9901-e33c608cabf5\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " Nov 02 20:19:33.047865 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.047845 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\" (UniqueName: \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\") pod \"34abd91e-2472-42ee-9901-e33c608cabf5\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " Nov 02 20:19:33.048123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.048096 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\") pod \"34abd91e-2472-42ee-9901-e33c608cabf5\" (UID: \"34abd91e-2472-42ee-9901-e33c608cabf5\") " Nov 02 20:19:33.053293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.053245 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca" pod "34abd91e-2472-42ee-9901-e33c608cabf5" (UID: "34abd91e-2472-42ee-9901-e33c608cabf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:33.091993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.091891 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls" pod "34abd91e-2472-42ee-9901-e33c608cabf5" (UID: "34abd91e-2472-42ee-9901-e33c608cabf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:33.094222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.094188 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs" pod "34abd91e-2472-42ee-9901-e33c608cabf5" (UID: "34abd91e-2472-42ee-9901-e33c608cabf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:33.094544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.094518 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk" pod "34abd91e-2472-42ee-9901-e33c608cabf5" (UID: "34abd91e-2472-42ee-9901-e33c608cabf5"). InnerVolumeSpecName "kube-api-access-k6hdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:33.149831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.149797 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hdk\" (UniqueName: \"kubernetes.io/projected/34abd91e-2472-42ee-9901-e33c608cabf5-kube-api-access-k6hdk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:33.149831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.149832 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34abd91e-2472-42ee-9901-e33c608cabf5-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:33.150025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.149845 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:33.150025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.149853 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34abd91e-2472-42ee-9901-e33c608cabf5-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:33.341086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.341054 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:19:33.342180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.342164 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:19:33.537034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.537004 5136 generic.go:356] "Generic (PLEG): container finished" podID="34abd91e-2472-42ee-9901-e33c608cabf5" containerID="665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c" exitCode=0 Nov 02 20:19:33.537214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.537050 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" Nov 02 20:19:33.537214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.537100 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" event={"ID":"34abd91e-2472-42ee-9901-e33c608cabf5","Type":"ContainerDied","Data":"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c"} Nov 02 20:19:33.537214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.537154 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz" event={"ID":"34abd91e-2472-42ee-9901-e33c608cabf5","Type":"ContainerDied","Data":"ceb8d66077e2594a8a239c88b2a941e704e6aa64552a9c6392bc5e84cd420aaa"} Nov 02 20:19:33.537214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.537182 5136 scope.go:122] "RemoveContainer" containerID="665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c" Nov 02 20:19:33.548099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.548076 5136 scope.go:122] "RemoveContainer" containerID="665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c" Nov 02 20:19:33.548417 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:33.548392 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c\": container with ID starting with 665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c not found: ID does not exist" containerID="665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c" Nov 02 20:19:33.548523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.548424 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c"} err="failed to get container status \"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c\": rpc error: code = NotFound desc = could not find container \"665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c\": container with ID starting with 665145880d9b865fc9523a3879d4fbb62c5d14548b81bc1258dd8ec4b8443a0c not found: ID does not exist" Nov 02 20:19:33.669863 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.669829 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:19:33.672650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:33.672623 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40aecd-59cc9fd9ff-8s6pz"] Nov 02 20:19:34.547305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:34.547273 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34abd91e-2472-42ee-9901-e33c608cabf5" path="/var/lib/kubelet/pods/34abd91e-2472-42ee-9901-e33c608cabf5/volumes" Nov 02 20:19:38.737037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.737001 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:19:38.737550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.737254 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" containerID="cri-o://fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153" gracePeriod=30 Nov 02 20:19:38.741240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.741208 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:19:38.741452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.741430 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" containerID="cri-o://99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7" gracePeriod=30 Nov 02 20:19:38.782221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.782190 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:19:38.782385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:38.782367 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" containerID="cri-o://239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b" gracePeriod=30 Nov 02 20:19:44.084399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.084364 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:19:44.108963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.108935 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:19:44.112700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.112676 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:19:44.230559 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230522 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230580 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230617 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230660 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230879 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230926 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.230974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.230967 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231018 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231062 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231065 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231122 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231377 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231393 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.231484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231455 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231494 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231697 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231758 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231765 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231798 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231859 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231894 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231938 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.232000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.231948 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232007 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232051 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232104 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232159 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232196 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232227 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\") pod \"32754173-8f0a-4223-87d3-dbc285464b8d\" (UID: \"32754173-8f0a-4223-87d3-dbc285464b8d\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232258 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232286 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\") pod \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\" (UID: \"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232328 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\") pod \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\" (UID: \"a1d6a818-2448-4dd1-b74e-97d1d73b3591\") " Nov 02 20:19:44.232541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.232393 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.233095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233056 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233078 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233091 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233104 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233119 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233132 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.233480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233351 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.233545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233524 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.233656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233637 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.233780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233717 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.233780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233770 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "kube-api-access-6vgpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.233917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233769 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.233917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233717 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.234030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233981 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.234030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.233994 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.234145 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.234036 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.234206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.234185 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.234267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.234201 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.234741 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.234674 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.235298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.235210 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.235384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.235300 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.236167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.236121 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.236289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.236266 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.236390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.236368 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.236886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.236869 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g" pod "a1d6a818-2448-4dd1-b74e-97d1d73b3591" (UID: "a1d6a818-2448-4dd1-b74e-97d1d73b3591"). InnerVolumeSpecName "kube-api-access-96v4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.237172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.237153 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v" pod "32754173-8f0a-4223-87d3-dbc285464b8d" (UID: "32754173-8f0a-4223-87d3-dbc285464b8d"). InnerVolumeSpecName "kube-api-access-nnh5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.237334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.237321 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token" pod "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" (UID: "ffd542ec-e3da-42db-8ba6-0b3ba7a4421f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.333978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333898 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.333978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333927 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnh5v\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-kube-api-access-nnh5v\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.333978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333942 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.333978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333955 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32754173-8f0a-4223-87d3-dbc285464b8d-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.333978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333968 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333981 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.333994 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334009 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32754173-8f0a-4223-87d3-dbc285464b8d-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334021 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334032 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334045 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334056 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334071 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgpn\" (UniqueName: \"kubernetes.io/projected/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-kube-api-access-6vgpn\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334085 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334098 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96v4g\" (UniqueName: \"kubernetes.io/projected/a1d6a818-2448-4dd1-b74e-97d1d73b3591-kube-api-access-96v4g\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334110 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334123 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32754173-8f0a-4223-87d3-dbc285464b8d-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334138 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334152 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32754173-8f0a-4223-87d3-dbc285464b8d-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334166 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.334292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.334179 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d6a818-2448-4dd1-b74e-97d1d73b3591-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.638026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.637950 5136 generic.go:356] "Generic (PLEG): container finished" podID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerID="fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153" exitCode=0 Nov 02 20:19:44.638026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.638020 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" Nov 02 20:19:44.638215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.638022 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" event={"ID":"a1d6a818-2448-4dd1-b74e-97d1d73b3591","Type":"ContainerDied","Data":"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153"} Nov 02 20:19:44.638215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.638084 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7" event={"ID":"a1d6a818-2448-4dd1-b74e-97d1d73b3591","Type":"ContainerDied","Data":"096ced5bb405b2689825afca759ee9202f5743c8b14feda7a9bc7dd17e00752b"} Nov 02 20:19:44.638215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.638115 5136 scope.go:122] "RemoveContainer" containerID="fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153" Nov 02 20:19:44.639635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.639613 5136 generic.go:356] "Generic (PLEG): container finished" podID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerID="99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7" exitCode=0 Nov 02 20:19:44.639759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.639682 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" event={"ID":"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f","Type":"ContainerDied","Data":"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7"} Nov 02 20:19:44.639759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.639706 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" Nov 02 20:19:44.639759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.639749 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd" event={"ID":"ffd542ec-e3da-42db-8ba6-0b3ba7a4421f","Type":"ContainerDied","Data":"32739d7aa21768177e1f6a0710cbd273cd5ce0938f83c0667b121a5340f7b7b2"} Nov 02 20:19:44.641746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.641711 5136 generic.go:356] "Generic (PLEG): container finished" podID="32754173-8f0a-4223-87d3-dbc285464b8d" containerID="239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b" exitCode=0 Nov 02 20:19:44.641850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.641758 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" event={"ID":"32754173-8f0a-4223-87d3-dbc285464b8d","Type":"ContainerDied","Data":"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b"} Nov 02 20:19:44.641850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.641792 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" Nov 02 20:19:44.641850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.641803 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s" event={"ID":"32754173-8f0a-4223-87d3-dbc285464b8d","Type":"ContainerDied","Data":"9161466b0408e05110fa00ae988074966d0020623e3c25e4a83c7887e8051482"} Nov 02 20:19:44.649486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.649472 5136 scope.go:122] "RemoveContainer" containerID="fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153" Nov 02 20:19:44.649771 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:44.649754 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153\": container with ID starting with fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153 not found: ID does not exist" containerID="fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153" Nov 02 20:19:44.649824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.649777 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153"} err="failed to get container status \"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153\": rpc error: code = NotFound desc = could not find container \"fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153\": container with ID starting with fccc1448c0f25a1a31365a868d0e00158afe5af98a5ecd0db3083e913cdfb153 not found: ID does not exist" Nov 02 20:19:44.649824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.649791 5136 scope.go:122] "RemoveContainer" containerID="99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7" Nov 02 20:19:44.658194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.658180 5136 scope.go:122] "RemoveContainer" containerID="99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7" Nov 02 20:19:44.658432 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:44.658417 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7\": container with ID starting with 99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7 not found: ID does not exist" containerID="99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7" Nov 02 20:19:44.658489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.658437 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7"} err="failed to get container status \"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7\": rpc error: code = NotFound desc = could not find container \"99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7\": container with ID starting with 99699c3195c6d14aa3861e2b82d069399ee9fa030f70e9ad6e4bad32d5fa45c7 not found: ID does not exist" Nov 02 20:19:44.658489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.658452 5136 scope.go:122] "RemoveContainer" containerID="239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b" Nov 02 20:19:44.666456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.666438 5136 scope.go:122] "RemoveContainer" containerID="239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b" Nov 02 20:19:44.666725 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:44.666705 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b\": container with ID starting with 239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b not found: ID does not exist" containerID="239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b" Nov 02 20:19:44.666911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.666757 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b"} err="failed to get container status \"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b\": rpc error: code = NotFound desc = could not find container \"239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b\": container with ID starting with 239372f68185a08cc064cea28c098fcdcdeb8976acf6550a5bf193efc94e7d5b not found: ID does not exist" Nov 02 20:19:44.795308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.795280 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:19:44.798297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.798273 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-8tl6s"] Nov 02 20:19:44.903877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.903798 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:19:44.906548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:44.906511 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wlcbd"] Nov 02 20:19:45.009607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:45.009580 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:19:45.016263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:45.016233 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-wdfc7"] Nov 02 20:19:46.547174 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:46.547144 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" path="/var/lib/kubelet/pods/32754173-8f0a-4223-87d3-dbc285464b8d/volumes" Nov 02 20:19:46.547843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:46.547822 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" path="/var/lib/kubelet/pods/a1d6a818-2448-4dd1-b74e-97d1d73b3591/volumes" Nov 02 20:19:46.548313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:19:46.548293 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" path="/var/lib/kubelet/pods/ffd542ec-e3da-42db-8ba6-0b3ba7a4421f/volumes" Nov 02 20:19:51.004495 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:19:51.004461 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:20:03.644376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.644348 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:20:03.644984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.644969 5136 cpu_manager.go:414] "containerMap: removing container" podUID="34abd91e-2472-42ee-9901-e33c608cabf5" containerName="maas-api" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.644985 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34abd91e-2472-42ee-9901-e33c608cabf5" containerName="maas-api" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.644994 5136 cpu_manager.go:414] "containerMap: removing container" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.644999 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645020 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645025 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645033 5136 cpu_manager.go:414] "containerMap: removing container" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" Nov 02 20:20:03.645042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645038 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645047 5136 cpu_manager.go:414] "containerMap: removing container" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645051 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645071 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645076 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645156 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d6a818-2448-4dd1-b74e-97d1d73b3591" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645165 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54ea449-0c01-48ec-91b2-ad42b3830434" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645172 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="659b69c2-464c-4ed8-a7c2-8178fc81a723" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645180 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd542ec-e3da-42db-8ba6-0b3ba7a4421f" containerName="istio-proxy" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645188 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="34abd91e-2472-42ee-9901-e33c608cabf5" containerName="maas-api" Nov 02 20:20:03.645270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.645194 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="32754173-8f0a-4223-87d3-dbc285464b8d" containerName="istio-proxy" Nov 02 20:20:03.648847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.648819 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.698764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.698693 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3b8db2-openshift-default-dockercfg-4zn22\"" Nov 02 20:20:03.751051 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.751010 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:20:03.753077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753045 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753157 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753242 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753280 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhnlr\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753339 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753472 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753538 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753611 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.753792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.753770 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.880801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880601 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.880801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880703 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.880801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880764 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.880801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880802 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.881153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880827 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhnlr\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.881153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880886 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.881153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.880968 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.881153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.881027 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.881153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.881049 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.902815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.881573 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.902815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.882109 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.902815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.884897 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.902815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.885143 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.902815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.886013 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.943700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.943591 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.944469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.944411 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.947180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.947158 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.947359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.947301 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhnlr\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\") pod \"e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:03.963776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:03.963447 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:04.388781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.388724 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:20:04.389623 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:20:04.389557 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af8216c_b210_494f_a975_96c702c654f1.slice/crio-192d9b4ddc3f34cfb9b541761ab48fbcdd68f1622c3bbf6ff04a9b325cc9fc28 WatchSource:0}: Error finding container 192d9b4ddc3f34cfb9b541761ab48fbcdd68f1622c3bbf6ff04a9b325cc9fc28: Status 404 returned error can't find the container with id 192d9b4ddc3f34cfb9b541761ab48fbcdd68f1622c3bbf6ff04a9b325cc9fc28 Nov 02 20:20:04.392677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.392641 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:20:04.392799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.392708 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:20:04.392799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.392774 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:20:04.740221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.740182 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:20:04.744784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.744758 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.796286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.794161 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d3184c-serving-cert\"" Nov 02 20:20:04.796286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.796206 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:20:04.897257 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.897223 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.897452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.897379 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.897452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.897420 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.897586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.897504 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.911814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.911782 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerStarted","Data":"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85"} Nov 02 20:20:04.911973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.911844 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerStarted","Data":"192d9b4ddc3f34cfb9b541761ab48fbcdd68f1622c3bbf6ff04a9b325cc9fc28"} Nov 02 20:20:04.965281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.965253 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:04.998623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.998535 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.998623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.998580 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.998908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.998755 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.998969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.998899 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:04.999377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:04.999339 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:05.001202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.001181 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:05.001305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.001236 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:05.022102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.022063 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podStartSLOduration=2.022051131 podStartE2EDuration="2.022051131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:05.021602642 +0000 UTC m=+1888.628147143" watchObservedRunningTime="2026-11-02 20:20:05.022051131 +0000 UTC m=+1888.628595629" Nov 02 20:20:05.053121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.053090 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\") pod \"maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:05.060851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.060825 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:05.434213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.434188 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:20:05.435438 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:20:05.435405 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc834fa04_4ede_495c_92bb_58205ef7d7c1.slice/crio-b6ec08cf065455eaa80b32bb79c9e45e070ac347bec4c680cd028bb35162cd35 WatchSource:0}: Error finding container b6ec08cf065455eaa80b32bb79c9e45e070ac347bec4c680cd028bb35162cd35: Status 404 returned error can't find the container with id b6ec08cf065455eaa80b32bb79c9e45e070ac347bec4c680cd028bb35162cd35 Nov 02 20:20:05.919426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.919380 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" event={"ID":"c834fa04-4ede-495c-92bb-58205ef7d7c1","Type":"ContainerStarted","Data":"b6ec08cf065455eaa80b32bb79c9e45e070ac347bec4c680cd028bb35162cd35"} Nov 02 20:20:05.970348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:05.970317 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:06.934578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:06.934534 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" event={"ID":"c834fa04-4ede-495c-92bb-58205ef7d7c1","Type":"ContainerStarted","Data":"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98"} Nov 02 20:20:06.935106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:06.934647 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:06.935106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:06.934702 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:06.936264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:06.936240 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:20:07.043249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:07.043199 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" podStartSLOduration=2.147090445 podStartE2EDuration="3.043185942s" totalImagesPullingTime="896.095497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:07.042156087 +0000 UTC m=+1890.648700586" watchObservedRunningTime="2026-11-02 20:20:07.043185942 +0000 UTC m=+1890.649730441" Nov 02 20:20:12.946262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:12.946232 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:20:21.003993 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:21.003915 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:20:28.164471 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.164436 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:20:28.168558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.168538 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.218610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.218555 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-316e97-serving-cert\"" Nov 02 20:20:28.229592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.229556 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.229773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.229682 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.229773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.229752 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgmx\" (UniqueName: \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.229950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.229930 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.268628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.268599 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:20:28.331717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.331677 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.331920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.331761 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.331920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.331800 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgmx\" (UniqueName: \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.331920 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:28.331897 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-serving-cert: secret "maas-api-e2e-ipp-b-316e97-serving-cert" not found Nov 02 20:20:28.332052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.331923 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.332052 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:28.331972 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls podName:2fe679ca-4341-4281-9c9d-d5392bb2331d nodeName:}" failed. No retries permitted until 2026-11-02 20:20:28.831952002 +0000 UTC m=+1912.438496481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls") pod "maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" (UID: "2fe679ca-4341-4281-9c9d-d5392bb2331d") : secret "maas-api-e2e-ipp-b-316e97-serving-cert" not found Nov 02 20:20:28.332444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.332423 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.334475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.334452 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.389797 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.389766 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgmx\" (UniqueName: \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.839013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.838980 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:28.841380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:28.841355 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:29.082466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:29.082440 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:29.452838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:29.452808 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:20:29.453719 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:20:29.453684 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe679ca_4341_4281_9c9d_d5392bb2331d.slice/crio-b4abc2b6964960d0ed8388b787b8155e0b5c776610354e057185098f001a0b9b WatchSource:0}: Error finding container b4abc2b6964960d0ed8388b787b8155e0b5c776610354e057185098f001a0b9b: Status 404 returned error can't find the container with id b4abc2b6964960d0ed8388b787b8155e0b5c776610354e057185098f001a0b9b Nov 02 20:20:30.080639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:30.080578 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" event={"ID":"2fe679ca-4341-4281-9c9d-d5392bb2331d","Type":"ContainerStarted","Data":"b4abc2b6964960d0ed8388b787b8155e0b5c776610354e057185098f001a0b9b"} Nov 02 20:20:31.091440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:31.091394 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" event={"ID":"2fe679ca-4341-4281-9c9d-d5392bb2331d","Type":"ContainerStarted","Data":"141c98662e9f4c79bc783c0e37aac08006a8523600df44548a4bee14d32c1a90"} Nov 02 20:20:31.092144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:31.092112 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:31.202062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:31.202002 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" podStartSLOduration=2.38711289 podStartE2EDuration="3.201984354s" totalImagesPullingTime="814.871464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:31.200867622 +0000 UTC m=+1914.807412131" watchObservedRunningTime="2026-11-02 20:20:31.201984354 +0000 UTC m=+1914.808528853" Nov 02 20:20:37.102935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:37.102906 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:20:40.523641 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:40.523606 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:20:40.524065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:40.523833 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" containerID="cri-o://ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b" gracePeriod=30 Nov 02 20:20:42.541030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:42.540987 5136 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:20:42.541417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:42.541054 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:20:45.900745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.900708 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:20:45.953773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953722 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.953925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953803 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.953925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953825 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.953925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953874 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953929 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953953 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.953993 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954062 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954221 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\") pod \"cde30c9f-9364-470c-8321-a0508ec544bc\" (UID: \"cde30c9f-9364-470c-8321-a0508ec544bc\") " Nov 02 20:20:45.954281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954221 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:45.954281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954265 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:45.954434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954334 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:45.954494 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954437 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:45.954650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954630 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:45.954834 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954818 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:45.954911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954839 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:45.954911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954853 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde30c9f-9364-470c-8321-a0508ec544bc-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:45.954911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954866 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:45.954911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.954881 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:45.956486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.956464 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:45.956556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.956513 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:45.956556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.956538 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "kube-api-access-2dhfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:45.956638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:45.956622 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo" pod "cde30c9f-9364-470c-8321-a0508ec544bc" (UID: "cde30c9f-9364-470c-8321-a0508ec544bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:20:46.056022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.055987 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde30c9f-9364-470c-8321-a0508ec544bc-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:46.056022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.056018 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde30c9f-9364-470c-8321-a0508ec544bc-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:46.056022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.056028 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:46.056236 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.056038 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhfc\" (UniqueName: \"kubernetes.io/projected/cde30c9f-9364-470c-8321-a0508ec544bc-kube-api-access-2dhfc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:46.183814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.183715 5136 generic.go:356] "Generic (PLEG): container finished" podID="cde30c9f-9364-470c-8321-a0508ec544bc" containerID="ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b" exitCode=0 Nov 02 20:20:46.183814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.183799 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" event={"ID":"cde30c9f-9364-470c-8321-a0508ec544bc","Type":"ContainerDied","Data":"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b"} Nov 02 20:20:46.184006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.183853 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" event={"ID":"cde30c9f-9364-470c-8321-a0508ec544bc","Type":"ContainerDied","Data":"3c494a0eeb820e910f3a04ed0779cb07d923625d1cb2cb4ffdf298cd2aa4e8d9"} Nov 02 20:20:46.184006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.183879 5136 scope.go:122] "RemoveContainer" containerID="ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b" Nov 02 20:20:46.184006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.183808 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg" Nov 02 20:20:46.194140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.194121 5136 scope.go:122] "RemoveContainer" containerID="ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b" Nov 02 20:20:46.194404 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:46.194384 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b\": container with ID starting with ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b not found: ID does not exist" containerID="ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b" Nov 02 20:20:46.194452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.194413 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b"} err="failed to get container status \"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b\": rpc error: code = NotFound desc = could not find container \"ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b\": container with ID starting with ab420b9b8bafe453d6fb42a17cb606fa7c48b466fa7b114362e215a4a8bb3f0b not found: ID does not exist" Nov 02 20:20:46.343663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.343630 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:20:46.347270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.347244 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-dzpmg"] Nov 02 20:20:46.548462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:46.548430 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" path="/var/lib/kubelet/pods/cde30c9f-9364-470c-8321-a0508ec544bc/volumes" Nov 02 20:20:50.097097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.097068 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:20:50.097669 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.097654 5136 cpu_manager.go:414] "containerMap: removing container" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" Nov 02 20:20:50.097715 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.097670 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" Nov 02 20:20:50.097809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.097799 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde30c9f-9364-470c-8321-a0508ec544bc" containerName="istio-proxy" Nov 02 20:20:50.102705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.102685 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.154936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.154907 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:20:50.176612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.176577 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.176830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.176628 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s75t\" (UniqueName: \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.176963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.176938 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.177129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.177096 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.197972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.197950 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e87e9a-serving-cert\"" Nov 02 20:20:50.277931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.277886 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.278098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.277994 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.278098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.278025 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s75t\" (UniqueName: \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.278098 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:50.278028 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-serving-cert: secret "maas-api-e2e-ipp-a-e87e9a-serving-cert" not found Nov 02 20:20:50.278098 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:50.278096 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls podName:0944088f-62a5-429d-9f1d-ce06399029b5 nodeName:}" failed. No retries permitted until 2026-11-02 20:20:50.778080777 +0000 UTC m=+1934.384625254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls") pod "maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" (UID: "0944088f-62a5-429d-9f1d-ce06399029b5") : secret "maas-api-e2e-ipp-a-e87e9a-serving-cert" not found Nov 02 20:20:50.278260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.278113 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.278795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.278776 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.280334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.280316 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.332607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.332576 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s75t\" (UniqueName: \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.789303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.789269 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:50.791915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:50.791881 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:51.004017 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:20:51.003981 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:20:51.015998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:51.015973 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:51.383471 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:51.383444 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:20:51.384046 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:20:51.384006 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0944088f_62a5_429d_9f1d_ce06399029b5.slice/crio-3ca428a7edc53924ea23f5790841ecd94341e6e47afe110b7ac6226e428c9cff WatchSource:0}: Error finding container 3ca428a7edc53924ea23f5790841ecd94341e6e47afe110b7ac6226e428c9cff: Status 404 returned error can't find the container with id 3ca428a7edc53924ea23f5790841ecd94341e6e47afe110b7ac6226e428c9cff Nov 02 20:20:52.269496 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:52.269455 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" event={"ID":"0944088f-62a5-429d-9f1d-ce06399029b5","Type":"ContainerStarted","Data":"3ca428a7edc53924ea23f5790841ecd94341e6e47afe110b7ac6226e428c9cff"} Nov 02 20:20:53.277605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:53.277574 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" event={"ID":"0944088f-62a5-429d-9f1d-ce06399029b5","Type":"ContainerStarted","Data":"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7"} Nov 02 20:20:53.278039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:53.277647 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:20:53.387700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:53.387646 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" podStartSLOduration=2.197368479 podStartE2EDuration="3.387630022s" totalImagesPullingTime="1.190261543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:53.386555072 +0000 UTC m=+1936.993099619" watchObservedRunningTime="2026-11-02 20:20:53.387630022 +0000 UTC m=+1936.994174520" Nov 02 20:20:59.290439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:20:59.290409 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:21:21.003936 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:21:21.003900 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:21:45.613112 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.613076 5136 generic.go:356] "Generic (PLEG): container finished" podID="9af8216c-b210-494f-a975-96c702c654f1" containerID="80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85" exitCode=0 Nov 02 20:21:45.613665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.613152 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerDied","Data":"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85"} Nov 02 20:21:45.613665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.613519 5136 scope.go:122] "RemoveContainer" containerID="80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85" Nov 02 20:21:45.614962 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.614936 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:45.615038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.615005 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:45.615097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:45.615036 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:46.621398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:46.621315 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerStarted","Data":"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400"} Nov 02 20:21:46.965088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:46.965057 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:21:46.965254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:46.965192 5136 patch_prober.go:29] interesting pod/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:21:46.965301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:46.965247 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 20:21:47.965485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:47.965448 5136 patch_prober.go:29] interesting pod/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:21:47.965987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:47.965506 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 20:21:48.966121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:48.966084 5136 patch_prober.go:29] interesting pod/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:21:48.966553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:48.966142 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 20:21:49.965766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:49.965713 5136 patch_prober.go:29] interesting pod/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:21:49.965947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:49.965795 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 20:21:50.965519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:50.965485 5136 patch_prober.go:29] interesting pod/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:21:50.966029 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:50.965559 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 20:21:51.028288 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:21:51.028257 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:21:51.935426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:51.935395 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:21:51.970343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:51.970308 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:21:52.676290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:21:52.676264 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:22:21.004794 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:22:21.004741 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:22:51.004847 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:22:51.004808 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:23:21.004275 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:21.004194 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:23:24.310820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.310786 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:23:24.314390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.314367 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.412129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.412088 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-57e2cf-serving-cert\"" Nov 02 20:23:24.414936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.414902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzst4\" (UniqueName: \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.415090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.415012 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.415157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.415100 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.415216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.415177 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.415348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.415329 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:23:24.516458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.516424 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.516669 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.516489 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.516669 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.516549 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.516669 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:24.516622 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-serving-cert: secret "maas-api-e2e-ipp-a-57e2cf-serving-cert" not found Nov 02 20:23:24.516669 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.516664 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzst4\" (UniqueName: \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.516949 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:24.516707 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls podName:02713b1b-f196-4964-81d2-b4b004cf3767 nodeName:}" failed. No retries permitted until 2026-11-02 20:23:25.016684895 +0000 UTC m=+2088.623229387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls") pod "maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" (UID: "02713b1b-f196-4964-81d2-b4b004cf3767") : secret "maas-api-e2e-ipp-a-57e2cf-serving-cert" not found Nov 02 20:23:24.517286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.517267 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.519093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.519069 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:24.598045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:24.594569 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzst4\" (UniqueName: \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:25.030268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:25.030236 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:25.032632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:25.032607 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") pod \"maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:25.228782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:25.228721 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:25.594296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:25.594274 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:23:25.595894 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:23:25.595766 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02713b1b_f196_4964_81d2_b4b004cf3767.slice/crio-1e6c4cc7bdbc798c3ebafa17812ad97854a27b30038c6c4b45dc234ab4208c25 WatchSource:0}: Error finding container 1e6c4cc7bdbc798c3ebafa17812ad97854a27b30038c6c4b45dc234ab4208c25: Status 404 returned error can't find the container with id 1e6c4cc7bdbc798c3ebafa17812ad97854a27b30038c6c4b45dc234ab4208c25 Nov 02 20:23:25.597268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:25.597246 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:23:26.490789 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:26.490759 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" event={"ID":"02713b1b-f196-4964-81d2-b4b004cf3767","Type":"ContainerStarted","Data":"1e6c4cc7bdbc798c3ebafa17812ad97854a27b30038c6c4b45dc234ab4208c25"} Nov 02 20:23:27.499032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:27.498986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" event={"ID":"02713b1b-f196-4964-81d2-b4b004cf3767","Type":"ContainerStarted","Data":"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289"} Nov 02 20:23:27.499468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:27.499193 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:27.610565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:27.610509 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" podStartSLOduration=2.691957866 podStartE2EDuration="3.610495064s" totalImagesPullingTime="918.537198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:27.607702958 +0000 UTC m=+2091.214247456" watchObservedRunningTime="2026-11-02 20:23:27.610495064 +0000 UTC m=+2091.217039562" Nov 02 20:23:29.238450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.238419 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:23:29.238901 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.238613 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" podUID="519b73a7-30e2-4c4b-bd95-22d336541988" containerName="maas-api" containerID="cri-o://ad46f60823eb26ea09a193c92681f2e85dd3b4b5f96fa0d8b0ff15516ade22ae" gracePeriod=30 Nov 02 20:23:29.537359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.537331 5136 generic.go:356] "Generic (PLEG): container finished" podID="519b73a7-30e2-4c4b-bd95-22d336541988" containerID="ad46f60823eb26ea09a193c92681f2e85dd3b4b5f96fa0d8b0ff15516ade22ae" exitCode=0 Nov 02 20:23:29.537495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.537383 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" event={"ID":"519b73a7-30e2-4c4b-bd95-22d336541988","Type":"ContainerDied","Data":"ad46f60823eb26ea09a193c92681f2e85dd3b4b5f96fa0d8b0ff15516ade22ae"} Nov 02 20:23:29.560713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.560689 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:23:29.723786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.722256 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") pod \"519b73a7-30e2-4c4b-bd95-22d336541988\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " Nov 02 20:23:29.723786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.722313 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\") pod \"519b73a7-30e2-4c4b-bd95-22d336541988\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " Nov 02 20:23:29.723786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.722344 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\" (UniqueName: \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\") pod \"519b73a7-30e2-4c4b-bd95-22d336541988\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " Nov 02 20:23:29.723786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.722479 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\") pod \"519b73a7-30e2-4c4b-bd95-22d336541988\" (UID: \"519b73a7-30e2-4c4b-bd95-22d336541988\") " Nov 02 20:23:29.756546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.756506 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca" pod "519b73a7-30e2-4c4b-bd95-22d336541988" (UID: "519b73a7-30e2-4c4b-bd95-22d336541988"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:29.802825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.793879 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs" pod "519b73a7-30e2-4c4b-bd95-22d336541988" (UID: "519b73a7-30e2-4c4b-bd95-22d336541988"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:29.804186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.804084 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls" pod "519b73a7-30e2-4c4b-bd95-22d336541988" (UID: "519b73a7-30e2-4c4b-bd95-22d336541988"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:29.804186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.804133 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2" pod "519b73a7-30e2-4c4b-bd95-22d336541988" (UID: "519b73a7-30e2-4c4b-bd95-22d336541988"). InnerVolumeSpecName "kube-api-access-zcrz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:29.824587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.824560 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.824587 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.824587 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519b73a7-30e2-4c4b-bd95-22d336541988-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.824780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.824597 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519b73a7-30e2-4c4b-bd95-22d336541988-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.824780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:29.824606 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrz2\" (UniqueName: \"kubernetes.io/projected/519b73a7-30e2-4c4b-bd95-22d336541988-kube-api-access-zcrz2\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:30.558522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:30.558493 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" Nov 02 20:23:30.559366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:30.559340 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq" event={"ID":"519b73a7-30e2-4c4b-bd95-22d336541988","Type":"ContainerDied","Data":"5ca062e7dec62661029ad00137315a1905c1bfa1519154f8510fb19eeabac231"} Nov 02 20:23:30.559421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:30.559395 5136 scope.go:122] "RemoveContainer" containerID="ad46f60823eb26ea09a193c92681f2e85dd3b4b5f96fa0d8b0ff15516ade22ae" Nov 02 20:23:30.736875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:30.736841 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:23:30.784537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:30.784505 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a999d7-8447568b89-qrbfq"] Nov 02 20:23:32.548138 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:32.548092 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519b73a7-30e2-4c4b-bd95-22d336541988" path="/var/lib/kubelet/pods/519b73a7-30e2-4c4b-bd95-22d336541988/volumes" Nov 02 20:23:33.482257 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:33.482225 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:23:33.482444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:33.482420 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" containerID="cri-o://f785e58772ea1fd69ef72ccb1ba05f02ee9c545d486b085fc93483d7c76ebf0a" gracePeriod=30 Nov 02 20:23:33.512286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:33.512264 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:23:38.671465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.671426 5136 generic.go:356] "Generic (PLEG): container finished" podID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerID="f785e58772ea1fd69ef72ccb1ba05f02ee9c545d486b085fc93483d7c76ebf0a" exitCode=0 Nov 02 20:23:38.671854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.671616 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerDied","Data":"f785e58772ea1fd69ef72ccb1ba05f02ee9c545d486b085fc93483d7c76ebf0a"} Nov 02 20:23:38.671854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.671678 5136 scope.go:122] "RemoveContainer" containerID="b9e30b52b61bb92c12676b615867e1b18dd716f7b01fe2e52f01c213237f99d5" Nov 02 20:23:38.747378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.747320 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:23:38.791647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791619 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.791820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791664 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.791820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791781 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.791935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791842 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.791935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791890 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.791935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791920 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.792116 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.791958 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.792116 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792003 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.792228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792182 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\") pod \"8e6fa59c-2b08-493e-8bf3-e6394f774236\" (UID: \"8e6fa59c-2b08-493e-8bf3-e6394f774236\") " Nov 02 20:23:38.792429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792160 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:38.792429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792200 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:38.792429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792303 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:38.792429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792415 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:38.792757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792512 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:38.792855 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792829 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6fa59c-2b08-493e-8bf3-e6394f774236-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.792981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792859 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.792981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792878 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.792981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792892 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.792981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.792907 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.794018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.793992 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:23:38.794136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.794029 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:38.794136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.794087 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "kube-api-access-bdjgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:38.794254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.794231 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token" pod "8e6fa59c-2b08-493e-8bf3-e6394f774236" (UID: "8e6fa59c-2b08-493e-8bf3-e6394f774236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:38.893658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.893633 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.893658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.893658 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.893859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.893667 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjgk\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-kube-api-access-bdjgk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:38.893859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:38.893678 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6fa59c-2b08-493e-8bf3-e6394f774236-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:39.679262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:39.679232 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" Nov 02 20:23:39.679721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:39.679287 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q" event={"ID":"8e6fa59c-2b08-493e-8bf3-e6394f774236","Type":"ContainerDied","Data":"44d1b312a4911a19037eaf5acb5cdbdd06a9381afbb55dcff0aca8b751e05410"} Nov 02 20:23:39.679721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:39.679338 5136 scope.go:122] "RemoveContainer" containerID="f785e58772ea1fd69ef72ccb1ba05f02ee9c545d486b085fc93483d7c76ebf0a" Nov 02 20:23:39.884143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:39.884077 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:23:39.932389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:39.932156 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a999d7-openshift-default-84498c4589-lgj8q"] Nov 02 20:23:40.547959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:40.547927 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" path="/var/lib/kubelet/pods/8e6fa59c-2b08-493e-8bf3-e6394f774236/volumes" Nov 02 20:23:42.106753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:42.106242 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:23:42.106753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:42.106535 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" containerID="cri-o://5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4" gracePeriod=30 Nov 02 20:23:42.779124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:42.779066 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:23:42.779324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:42.779150 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:23:46.394491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.394453 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:23:46.394902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.394664 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" podUID="c834fa04-4ede-495c-92bb-58205ef7d7c1" containerName="maas-api" containerID="cri-o://2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98" gracePeriod=30 Nov 02 20:23:46.701491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.701467 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:23:46.728516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.728490 5136 generic.go:356] "Generic (PLEG): container finished" podID="c834fa04-4ede-495c-92bb-58205ef7d7c1" containerID="2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98" exitCode=0 Nov 02 20:23:46.728651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.728536 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" Nov 02 20:23:46.728651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.728565 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" event={"ID":"c834fa04-4ede-495c-92bb-58205ef7d7c1","Type":"ContainerDied","Data":"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98"} Nov 02 20:23:46.728651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.728617 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c" event={"ID":"c834fa04-4ede-495c-92bb-58205ef7d7c1","Type":"ContainerDied","Data":"b6ec08cf065455eaa80b32bb79c9e45e070ac347bec4c680cd028bb35162cd35"} Nov 02 20:23:46.728651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.728638 5136 scope.go:122] "RemoveContainer" containerID="2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98" Nov 02 20:23:46.737718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.737701 5136 scope.go:122] "RemoveContainer" containerID="2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98" Nov 02 20:23:46.738051 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:46.738024 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98\": container with ID starting with 2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98 not found: ID does not exist" containerID="2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98" Nov 02 20:23:46.738131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.738056 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98"} err="failed to get container status \"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98\": rpc error: code = NotFound desc = could not find container \"2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98\": container with ID starting with 2f71142537d08267da86bf24a8b91363f1b9874d1ca148f65313aac1f044cd98 not found: ID does not exist" Nov 02 20:23:46.824754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.821121 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\") pod \"c834fa04-4ede-495c-92bb-58205ef7d7c1\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " Nov 02 20:23:46.824754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.821222 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\") pod \"c834fa04-4ede-495c-92bb-58205ef7d7c1\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " Nov 02 20:23:46.824754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.821310 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\") pod \"c834fa04-4ede-495c-92bb-58205ef7d7c1\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " Nov 02 20:23:46.824754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.821401 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\") pod \"c834fa04-4ede-495c-92bb-58205ef7d7c1\" (UID: \"c834fa04-4ede-495c-92bb-58205ef7d7c1\") " Nov 02 20:23:46.824754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.824312 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca" pod "c834fa04-4ede-495c-92bb-58205ef7d7c1" (UID: "c834fa04-4ede-495c-92bb-58205ef7d7c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:46.840754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.833299 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49" pod "c834fa04-4ede-495c-92bb-58205ef7d7c1" (UID: "c834fa04-4ede-495c-92bb-58205ef7d7c1"). InnerVolumeSpecName "kube-api-access-kgv49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:46.904337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.902071 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs" pod "c834fa04-4ede-495c-92bb-58205ef7d7c1" (UID: "c834fa04-4ede-495c-92bb-58205ef7d7c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:46.904337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.902344 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls" pod "c834fa04-4ede-495c-92bb-58205ef7d7c1" (UID: "c834fa04-4ede-495c-92bb-58205ef7d7c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:46.923527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.923499 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.923527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.923525 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c834fa04-4ede-495c-92bb-58205ef7d7c1-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.923527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.923534 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/c834fa04-4ede-495c-92bb-58205ef7d7c1-kube-api-access-kgv49\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.923776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:46.923544 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c834fa04-4ede-495c-92bb-58205ef7d7c1-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.147718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.147692 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:23:47.151536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.151490 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3184c-5d98854c66-vwv7c"] Nov 02 20:23:47.372440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.372415 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:23:47.564634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564601 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564766 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564803 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564865 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564912 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.564944 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565022 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565079 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565112 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565171 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\") pod \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\" (UID: \"89f7d876-5c3f-4ea0-9fae-cd555d11a60c\") " Nov 02 20:23:47.565654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565268 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:47.565654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565352 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565654 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565798 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565831 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565851 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565866 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.565877 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.565880 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.567529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.567501 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "kube-api-access-fkfll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:47.567644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.567591 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:47.567702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.567679 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:23:47.567771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.567687 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token" pod "89f7d876-5c3f-4ea0-9fae-cd555d11a60c" (UID: "89f7d876-5c3f-4ea0-9fae-cd555d11a60c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:47.667109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.667019 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.667109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.667050 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfll\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-kube-api-access-fkfll\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.667109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.667064 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.667109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.667076 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.667109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.667089 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f7d876-5c3f-4ea0-9fae-cd555d11a60c-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:47.738508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.738469 5136 generic.go:356] "Generic (PLEG): container finished" podID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerID="5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4" exitCode=0 Nov 02 20:23:47.738703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.738521 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" Nov 02 20:23:47.738703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.738545 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" event={"ID":"89f7d876-5c3f-4ea0-9fae-cd555d11a60c","Type":"ContainerDied","Data":"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4"} Nov 02 20:23:47.738703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.738603 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf" event={"ID":"89f7d876-5c3f-4ea0-9fae-cd555d11a60c","Type":"ContainerDied","Data":"e98f08bdb719289dc9a704cbf416764dbb3b4f97715621fea08805ac18346703"} Nov 02 20:23:47.738703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.738624 5136 scope.go:122] "RemoveContainer" containerID="5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4" Nov 02 20:23:47.753481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.753461 5136 scope.go:122] "RemoveContainer" containerID="5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4" Nov 02 20:23:47.753810 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:47.753784 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4\": container with ID starting with 5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4 not found: ID does not exist" containerID="5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4" Nov 02 20:23:47.753881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.753823 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4"} err="failed to get container status \"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4\": rpc error: code = NotFound desc = could not find container \"5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4\": container with ID starting with 5ad35194fe6fd4d60ebf4679710207fa394a1f2f05efee9e54fde78e7fb49ac4 not found: ID does not exist" Nov 02 20:23:47.894851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.894821 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:23:47.898683 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:47.898649 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-jjphf"] Nov 02 20:23:48.547175 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:48.547135 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" path="/var/lib/kubelet/pods/89f7d876-5c3f-4ea0-9fae-cd555d11a60c/volumes" Nov 02 20:23:48.547666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:48.547644 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c834fa04-4ede-495c-92bb-58205ef7d7c1" path="/var/lib/kubelet/pods/c834fa04-4ede-495c-92bb-58205ef7d7c1/volumes" Nov 02 20:23:51.003897 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:23:51.003864 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:23:54.677667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:54.677635 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:23:54.678170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:54.677874 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" podUID="e274626b-2cba-47f3-8de6-dcf5d49b9870" containerName="maas-api" containerID="cri-o://df3f6f46ec6241a73a710b79eeea6de9851bf5e4257fc8d6c75036381f67bcaf" gracePeriod=30 Nov 02 20:23:54.816534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:54.816496 5136 generic.go:356] "Generic (PLEG): container finished" podID="e274626b-2cba-47f3-8de6-dcf5d49b9870" containerID="df3f6f46ec6241a73a710b79eeea6de9851bf5e4257fc8d6c75036381f67bcaf" exitCode=0 Nov 02 20:23:54.816702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:54.816561 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" event={"ID":"e274626b-2cba-47f3-8de6-dcf5d49b9870","Type":"ContainerDied","Data":"df3f6f46ec6241a73a710b79eeea6de9851bf5e4257fc8d6c75036381f67bcaf"} Nov 02 20:23:55.004592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.004567 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:23:55.071820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.071790 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\" (UniqueName: \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\") pod \"e274626b-2cba-47f3-8de6-dcf5d49b9870\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " Nov 02 20:23:55.071983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.071917 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\") pod \"e274626b-2cba-47f3-8de6-dcf5d49b9870\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " Nov 02 20:23:55.071983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.071956 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\") pod \"e274626b-2cba-47f3-8de6-dcf5d49b9870\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " Nov 02 20:23:55.072099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.071987 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") pod \"e274626b-2cba-47f3-8de6-dcf5d49b9870\" (UID: \"e274626b-2cba-47f3-8de6-dcf5d49b9870\") " Nov 02 20:23:55.072385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.072354 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca" pod "e274626b-2cba-47f3-8de6-dcf5d49b9870" (UID: "e274626b-2cba-47f3-8de6-dcf5d49b9870"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:55.072719 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.072692 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e274626b-2cba-47f3-8de6-dcf5d49b9870-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:55.074162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.074132 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls" pod "e274626b-2cba-47f3-8de6-dcf5d49b9870" (UID: "e274626b-2cba-47f3-8de6-dcf5d49b9870"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:55.074287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.074159 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs" pod "e274626b-2cba-47f3-8de6-dcf5d49b9870" (UID: "e274626b-2cba-47f3-8de6-dcf5d49b9870"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:55.074287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.074215 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz" pod "e274626b-2cba-47f3-8de6-dcf5d49b9870" (UID: "e274626b-2cba-47f3-8de6-dcf5d49b9870"). InnerVolumeSpecName "kube-api-access-dk2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:55.204553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.204455 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2bz\" (UniqueName: \"kubernetes.io/projected/e274626b-2cba-47f3-8de6-dcf5d49b9870-kube-api-access-dk2bz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:55.204553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.204494 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:55.204553 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.204510 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e274626b-2cba-47f3-8de6-dcf5d49b9870-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:55.824266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.824225 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" event={"ID":"e274626b-2cba-47f3-8de6-dcf5d49b9870","Type":"ContainerDied","Data":"c0258dc0eea1334155c497ecc32750b00021486afad48326d27761c772037b2a"} Nov 02 20:23:55.824266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.824251 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q" Nov 02 20:23:55.824761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:55.824295 5136 scope.go:122] "RemoveContainer" containerID="df3f6f46ec6241a73a710b79eeea6de9851bf5e4257fc8d6c75036381f67bcaf" Nov 02 20:23:56.034348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:56.034310 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:23:56.049046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:56.048372 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-296ae9-55bc678bfd-sd97q"] Nov 02 20:23:56.549825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:56.549796 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e274626b-2cba-47f3-8de6-dcf5d49b9870" path="/var/lib/kubelet/pods/e274626b-2cba-47f3-8de6-dcf5d49b9870/volumes" Nov 02 20:23:58.651781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.651746 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:23:58.652409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652393 5136 cpu_manager.go:414] "containerMap: removing container" podUID="519b73a7-30e2-4c4b-bd95-22d336541988" containerName="maas-api" Nov 02 20:23:58.652462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652410 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519b73a7-30e2-4c4b-bd95-22d336541988" containerName="maas-api" Nov 02 20:23:58.652462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652432 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652437 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652452 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652461 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652472 5136 cpu_manager.go:414] "containerMap: removing container" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652477 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652492 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c834fa04-4ede-495c-92bb-58205ef7d7c1" containerName="maas-api" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652497 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c834fa04-4ede-495c-92bb-58205ef7d7c1" containerName="maas-api" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652507 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e274626b-2cba-47f3-8de6-dcf5d49b9870" containerName="maas-api" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652511 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e274626b-2cba-47f3-8de6-dcf5d49b9870" containerName="maas-api" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652594 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e274626b-2cba-47f3-8de6-dcf5d49b9870" containerName="maas-api" Nov 02 20:23:58.652616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652610 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="519b73a7-30e2-4c4b-bd95-22d336541988" containerName="maas-api" Nov 02 20:23:58.652876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652620 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652627 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6fa59c-2b08-493e-8bf3-e6394f774236" containerName="istio-proxy" Nov 02 20:23:58.652876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652634 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f7d876-5c3f-4ea0-9fae-cd555d11a60c" containerName="istio-proxy" Nov 02 20:23:58.652876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.652642 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c834fa04-4ede-495c-92bb-58205ef7d7c1" containerName="maas-api" Nov 02 20:23:58.658680 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.658652 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.756516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.756480 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:23:58.760297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760264 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhmc\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760405 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760483 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760579 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760722 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760629 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760744 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760813 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760871 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760844 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.760921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.760905 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.762690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.762673 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ed4910-openshift-default-dockercfg-srlh4\"" Nov 02 20:23:58.868667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868589 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.868862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868677 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhmc\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.868862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868771 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.868862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868819 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.869054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868877 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.869054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868916 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.869054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.868977 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.869054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.869024 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.869278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.869063 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.870151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.869576 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.870311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.870204 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.870384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.870363 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.870761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.870697 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.871033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.871010 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.873369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.873348 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.873760 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.873744 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.922811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.922753 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.923255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.923237 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhmc\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\") pod \"e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:58.971228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:58.971203 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:59.048661 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.048629 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:23:59.048946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.048916 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" containerID="cri-o://0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65" gracePeriod=30 Nov 02 20:23:59.450062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.450032 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:23:59.450515 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:23:59.450490 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750cdea_5020_42e7_9760_993e101592cd.slice/crio-433f3d59d0d7170a6bc0f45fabfe1e4002388ba7c9957eb1e5f1117da0d1bdc2 WatchSource:0}: Error finding container 433f3d59d0d7170a6bc0f45fabfe1e4002388ba7c9957eb1e5f1117da0d1bdc2: Status 404 returned error can't find the container with id 433f3d59d0d7170a6bc0f45fabfe1e4002388ba7c9957eb1e5f1117da0d1bdc2 Nov 02 20:23:59.453329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.453294 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:23:59.453422 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.453362 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:23:59.453422 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.453391 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:23:59.876100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.876014 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerStarted","Data":"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe"} Nov 02 20:23:59.876100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.876078 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerStarted","Data":"433f3d59d0d7170a6bc0f45fabfe1e4002388ba7c9957eb1e5f1117da0d1bdc2"} Nov 02 20:23:59.971623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.971588 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:23:59.987077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:23:59.987032 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podStartSLOduration=1.987019428 podStartE2EDuration="1.987019428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:59.985696098 +0000 UTC m=+2123.592240625" watchObservedRunningTime="2026-11-02 20:23:59.987019428 +0000 UTC m=+2123.593563926" Nov 02 20:24:00.976921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:00.976892 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:24:01.892825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:01.892795 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:24:01.893800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:01.893781 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:24:03.343405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:03.343366 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:24:03.343859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:03.343425 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:24:04.309530 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.309506 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:24:04.337763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337673 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.337763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337712 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.337953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337778 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.337953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337841 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.337953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337895 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.337953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337919 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:04.338158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.337962 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.338158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338046 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:04.338158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338061 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:04.338158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338069 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.338158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338108 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.338418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338205 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\") pod \"28a352b3-8451-4503-9ef8-c599a21831eb\" (UID: \"28a352b3-8451-4503-9ef8-c599a21831eb\") " Nov 02 20:24:04.338418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338323 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:04.338418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338361 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:04.338714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338689 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a352b3-8451-4503-9ef8-c599a21831eb-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.338858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338719 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.338858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338749 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.338858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338764 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.338858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.338778 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.340307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.340285 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "kube-api-access-z7fqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:04.340636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.340612 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:04.340793 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.340769 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:04.340864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.340828 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo" pod "28a352b3-8451-4503-9ef8-c599a21831eb" (UID: "28a352b3-8451-4503-9ef8-c599a21831eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:24:04.440278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.440253 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a352b3-8451-4503-9ef8-c599a21831eb-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.440278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.440277 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a352b3-8451-4503-9ef8-c599a21831eb-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.440646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.440287 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7fqk\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-kube-api-access-z7fqk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.440646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.440298 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a352b3-8451-4503-9ef8-c599a21831eb-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:04.912354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.912322 5136 generic.go:356] "Generic (PLEG): container finished" podID="28a352b3-8451-4503-9ef8-c599a21831eb" containerID="0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65" exitCode=0 Nov 02 20:24:04.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.912355 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerDied","Data":"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65"} Nov 02 20:24:04.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.912374 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" Nov 02 20:24:04.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.912413 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq" event={"ID":"28a352b3-8451-4503-9ef8-c599a21831eb","Type":"ContainerDied","Data":"06efcba7fcb97ba0e808df9abbb851541d8b1b0ca0f9ab3a8e72f84a43d588c9"} Nov 02 20:24:04.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.912437 5136 scope.go:122] "RemoveContainer" containerID="0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65" Nov 02 20:24:04.921554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.921535 5136 scope.go:122] "RemoveContainer" containerID="69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef" Nov 02 20:24:04.929894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.929877 5136 scope.go:122] "RemoveContainer" containerID="0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65" Nov 02 20:24:04.930182 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:24:04.930163 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65\": container with ID starting with 0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65 not found: ID does not exist" containerID="0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65" Nov 02 20:24:04.930237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.930190 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65"} err="failed to get container status \"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65\": rpc error: code = NotFound desc = could not find container \"0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65\": container with ID starting with 0224a8078d5369c6710efb5336eea50910726fb59abaf9a349770c6a7a174d65 not found: ID does not exist" Nov 02 20:24:04.930237 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.930208 5136 scope.go:122] "RemoveContainer" containerID="69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef" Nov 02 20:24:04.930456 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:24:04.930436 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef\": container with ID starting with 69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef not found: ID does not exist" containerID="69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef" Nov 02 20:24:04.930510 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:04.930465 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef"} err="failed to get container status \"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef\": rpc error: code = NotFound desc = could not find container \"69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef\": container with ID starting with 69b022f1991cb4314676d1ee74160ba857994afe7afad6f367c98add17c176ef not found: ID does not exist" Nov 02 20:24:05.068291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:05.068258 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:24:05.115603 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:05.115566 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-296ae9-openshift-default-78b7b546bc-xfzqq"] Nov 02 20:24:06.351776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.351726 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:24:06.352393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352376 5136 cpu_manager.go:414] "containerMap: removing container" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.352433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352395 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.352433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352414 5136 cpu_manager.go:414] "containerMap: removing container" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.352433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352419 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.352541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352516 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.352541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.352530 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" containerName="istio-proxy" Nov 02 20:24:06.356190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.356167 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.407604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.407576 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:24:06.455753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.452104 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ed4910-dockercfg-4r7d8\"" Nov 02 20:24:06.455753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.452203 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ed4910\"" Nov 02 20:24:06.498969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.476530 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.547816 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.547785 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8jr\" (UniqueName: \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.548215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.518397 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:24:06.553900 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.553864 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.569603 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.566603 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a352b3-8451-4503-9ef8-c599a21831eb" path="/var/lib/kubelet/pods/28a352b3-8451-4503-9ef8-c599a21831eb/volumes" Nov 02 20:24:06.580368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.580337 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:24:06.649601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.649454 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.649601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.649575 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.649918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.649770 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8jr\" (UniqueName: \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.649918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.649877 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn7b\" (UniqueName: \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.650397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.650378 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.703977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.703946 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8jr\" (UniqueName: \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\") pod \"payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:06.751127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.751083 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.751279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.751194 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn7b\" (UniqueName: \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.751955 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.751933 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.801927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.801897 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:24:06.802140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.802103 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerName="maas-api" containerID="cri-o://141c98662e9f4c79bc783c0e37aac08006a8523600df44548a4bee14d32c1a90" gracePeriod=30 Nov 02 20:24:06.807185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.806126 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn7b\" (UniqueName: \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\") pod \"payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.873717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.873686 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:06.927930 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.927904 5136 generic.go:356] "Generic (PLEG): container finished" podID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerID="141c98662e9f4c79bc783c0e37aac08006a8523600df44548a4bee14d32c1a90" exitCode=0 Nov 02 20:24:06.928072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.927974 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" event={"ID":"2fe679ca-4341-4281-9c9d-d5392bb2331d","Type":"ContainerDied","Data":"141c98662e9f4c79bc783c0e37aac08006a8523600df44548a4bee14d32c1a90"} Nov 02 20:24:06.974540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:06.974502 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:07.179167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.179106 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:24:07.257385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.257272 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:24:07.260589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.260562 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\") pod \"2fe679ca-4341-4281-9c9d-d5392bb2331d\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " Nov 02 20:24:07.261470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.261445 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\" (UniqueName: \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\") pod \"2fe679ca-4341-4281-9c9d-d5392bb2331d\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " Nov 02 20:24:07.261665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.261650 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") pod \"2fe679ca-4341-4281-9c9d-d5392bb2331d\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " Nov 02 20:24:07.261817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.261803 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\") pod \"2fe679ca-4341-4281-9c9d-d5392bb2331d\" (UID: \"2fe679ca-4341-4281-9c9d-d5392bb2331d\") " Nov 02 20:24:07.263742 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.263701 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca" pod "2fe679ca-4341-4281-9c9d-d5392bb2331d" (UID: "2fe679ca-4341-4281-9c9d-d5392bb2331d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:07.273059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.266774 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx" pod "2fe679ca-4341-4281-9c9d-d5392bb2331d" (UID: "2fe679ca-4341-4281-9c9d-d5392bb2331d"). InnerVolumeSpecName "kube-api-access-fpgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:07.273059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.272765 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs" pod "2fe679ca-4341-4281-9c9d-d5392bb2331d" (UID: "2fe679ca-4341-4281-9c9d-d5392bb2331d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:07.273305 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:24:07.273275 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc646ffaf_d3bc_45a6_ae45_37b9c9ffd201.slice/crio-544b621b5cfb9063d51674b42c385e75495c802a907e464bf26559e7645d845f WatchSource:0}: Error finding container 544b621b5cfb9063d51674b42c385e75495c802a907e464bf26559e7645d845f: Status 404 returned error can't find the container with id 544b621b5cfb9063d51674b42c385e75495c802a907e464bf26559e7645d845f Nov 02 20:24:07.273398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.273369 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls" pod "2fe679ca-4341-4281-9c9d-d5392bb2331d" (UID: "2fe679ca-4341-4281-9c9d-d5392bb2331d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:07.363848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.363824 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpgmx\" (UniqueName: \"kubernetes.io/projected/2fe679ca-4341-4281-9c9d-d5392bb2331d-kube-api-access-fpgmx\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.363848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.363850 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.364191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.363859 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe679ca-4341-4281-9c9d-d5392bb2331d-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.364191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.363867 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe679ca-4341-4281-9c9d-d5392bb2331d-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.410134 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.410106 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:24:07.411329 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:24:07.411299 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c99c37_52fe_4dbf_907b_5f29c6b19f10.slice/crio-1fe117685ba2ad048d5f6f7c770b0abdd533dcfb0c884ede6ddde4a00520c03e WatchSource:0}: Error finding container 1fe117685ba2ad048d5f6f7c770b0abdd533dcfb0c884ede6ddde4a00520c03e: Status 404 returned error can't find the container with id 1fe117685ba2ad048d5f6f7c770b0abdd533dcfb0c884ede6ddde4a00520c03e Nov 02 20:24:07.946240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.946201 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" event={"ID":"73c99c37-52fe-4dbf-907b-5f29c6b19f10","Type":"ContainerStarted","Data":"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9"} Nov 02 20:24:07.946484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.946462 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" event={"ID":"73c99c37-52fe-4dbf-907b-5f29c6b19f10","Type":"ContainerStarted","Data":"1fe117685ba2ad048d5f6f7c770b0abdd533dcfb0c884ede6ddde4a00520c03e"} Nov 02 20:24:07.947407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.947383 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:07.949196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.949168 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" Nov 02 20:24:07.949291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.949219 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" event={"ID":"2fe679ca-4341-4281-9c9d-d5392bb2331d","Type":"ContainerDied","Data":"b4abc2b6964960d0ed8388b787b8155e0b5c776610354e057185098f001a0b9b"} Nov 02 20:24:07.949291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.949278 5136 scope.go:122] "RemoveContainer" containerID="141c98662e9f4c79bc783c0e37aac08006a8523600df44548a4bee14d32c1a90" Nov 02 20:24:07.951761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.951739 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" event={"ID":"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201","Type":"ContainerStarted","Data":"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793"} Nov 02 20:24:07.951924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.951907 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" event={"ID":"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201","Type":"ContainerStarted","Data":"544b621b5cfb9063d51674b42c385e75495c802a907e464bf26559e7645d845f"} Nov 02 20:24:07.952725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:07.952705 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:08.055574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:08.055483 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" podStartSLOduration=2.055470685 podStartE2EDuration="2.055470685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:08.053715279 +0000 UTC m=+2131.660259776" watchObservedRunningTime="2026-11-02 20:24:08.055470685 +0000 UTC m=+2131.662015180" Nov 02 20:24:08.163471 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:08.163342 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" podStartSLOduration=2.163324501 podStartE2EDuration="2.163324501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:08.162092787 +0000 UTC m=+2131.768637286" watchObservedRunningTime="2026-11-02 20:24:08.163324501 +0000 UTC m=+2131.769868999" Nov 02 20:24:08.268138 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:08.268103 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:24:08.271763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:08.271720 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g"] Nov 02 20:24:08.547351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:08.547316 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" path="/var/lib/kubelet/pods/2fe679ca-4341-4281-9c9d-d5392bb2331d/volumes" Nov 02 20:24:10.100384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:10.100337 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-316e97-68d9c5b6-g5x8g" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.109:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:24:15.233621 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.233587 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:24:15.234367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.234347 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerName="maas-api" Nov 02 20:24:15.234420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.234371 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerName="maas-api" Nov 02 20:24:15.234547 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.234534 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe679ca-4341-4281-9c9d-d5392bb2331d" containerName="maas-api" Nov 02 20:24:15.238140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.238122 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.301751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.292571 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b028e6-openshift-default-dockercfg-n727n\"" Nov 02 20:24:15.322805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.322766 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323004 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.322982 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323092 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323245 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323292 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323320 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323553 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323662 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323639 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfr8\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.323748 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.323703 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.396326 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.396293 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.428890 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429052 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429085 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429109 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429177 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429215 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfr8\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429242 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429268 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429315 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.431791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.429664 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.475239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.432451 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.475239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.432696 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.475239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.437324 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.475239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.437654 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.477052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.477020 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.479298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.479268 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.486778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.485911 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfr8\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.486778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.486229 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\") pod \"e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.551803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.551773 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:15.974378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:15.974337 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:24:16.003381 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:24:16.003346 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fa5caf_2a3a_4b54_9e2c_bd5d28cfd488.slice/crio-8f537c936a7eaa99b6be596e10e0723ea44c622ab4f654d967aeddc8accfab84 WatchSource:0}: Error finding container 8f537c936a7eaa99b6be596e10e0723ea44c622ab4f654d967aeddc8accfab84: Status 404 returned error can't find the container with id 8f537c936a7eaa99b6be596e10e0723ea44c622ab4f654d967aeddc8accfab84 Nov 02 20:24:16.005815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:16.005779 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:16.005897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:16.005853 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:16.005897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:16.005881 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:17.009898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:17.009859 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerStarted","Data":"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7"} Nov 02 20:24:17.010291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:17.009927 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerStarted","Data":"8f537c936a7eaa99b6be596e10e0723ea44c622ab4f654d967aeddc8accfab84"} Nov 02 20:24:17.170824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:17.170773 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podStartSLOduration=2.170759668 podStartE2EDuration="2.170759668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:17.168154342 +0000 UTC m=+2140.774698843" watchObservedRunningTime="2026-11-02 20:24:17.170759668 +0000 UTC m=+2140.777304166" Nov 02 20:24:17.551927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:17.551898 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:17.558254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:17.558231 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:18.016472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:18.016433 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:18.017268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:18.017248 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:24:19.967429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:19.967395 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:24:19.967947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:24:19.967645 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:24:21.008135 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:24:21.008105 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:24:51.005420 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:24:51.005333 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:25:07.345541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.345508 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:25:07.349993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.349965 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.399435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.399400 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ed4910-serving-cert\"" Nov 02 20:25:07.437819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.437780 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.438013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.437908 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29g7g\" (UniqueName: \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.438086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.438060 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.438214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.438198 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.449551 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.449526 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:25:07.539103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.539064 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.539313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.539128 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29g7g\" (UniqueName: \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.539313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.539208 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.539313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.539302 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.539482 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:07.539457 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-serving-cert: secret "maas-api-e2e-praxis-a-ed4910-serving-cert" not found Nov 02 20:25:07.539547 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:07.539536 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls podName:1280b5d6-7207-480f-a3d3-1da4974df00d nodeName:}" failed. No retries permitted until 2026-11-02 20:25:08.039513547 +0000 UTC m=+2191.646058022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls") pod "maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" (UID: "1280b5d6-7207-480f-a3d3-1da4974df00d") : secret "maas-api-e2e-praxis-a-ed4910-serving-cert" not found Nov 02 20:25:07.540022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.539997 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.541944 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.541921 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:07.597473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:07.597371 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29g7g\" (UniqueName: \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:08.047015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:08.046973 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:08.049506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:08.049482 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:08.264209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:08.264174 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:08.635072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:08.635038 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:25:08.636789 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:25:08.636762 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1280b5d6_7207_480f_a3d3_1da4974df00d.slice/crio-366ae752470574ada039cf721e04d32b37bf2968e29d78aee9bd12a13038ba30 WatchSource:0}: Error finding container 366ae752470574ada039cf721e04d32b37bf2968e29d78aee9bd12a13038ba30: Status 404 returned error can't find the container with id 366ae752470574ada039cf721e04d32b37bf2968e29d78aee9bd12a13038ba30 Nov 02 20:25:09.378845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:09.378794 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" event={"ID":"1280b5d6-7207-480f-a3d3-1da4974df00d","Type":"ContainerStarted","Data":"366ae752470574ada039cf721e04d32b37bf2968e29d78aee9bd12a13038ba30"} Nov 02 20:25:10.387266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:10.387230 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" event={"ID":"1280b5d6-7207-480f-a3d3-1da4974df00d","Type":"ContainerStarted","Data":"02316dccbbe72c8cc711ede8fcfb5865b105a383f15a6895221d44a0aaef9c29"} Nov 02 20:25:10.387771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:10.387330 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:10.515750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:10.515640 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" podStartSLOduration=2.655518803 podStartE2EDuration="3.515612117s" totalImagesPullingTime="860.093314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:10.514970249 +0000 UTC m=+2194.121514774" watchObservedRunningTime="2026-11-02 20:25:10.515612117 +0000 UTC m=+2194.122156658" Nov 02 20:25:16.399208 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:16.399179 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:25:21.003225 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:21.003187 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:25:21.666151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:21.666116 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:25:21.666371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:21.666348 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" podUID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" containerName="maas-api" containerID="cri-o://dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992" gracePeriod=30 Nov 02 20:25:21.991481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:21.991451 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:25:22.040782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.040716 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") pod \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " Nov 02 20:25:22.041274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.040849 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\") pod \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " Nov 02 20:25:22.041274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.040922 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\") pod \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " Nov 02 20:25:22.041274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.041002 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\" (UniqueName: \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\") pod \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\" (UID: \"96fb560a-8ea2-4852-b725-a2f9ce6c13ce\") " Nov 02 20:25:22.041447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.041265 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca" pod "96fb560a-8ea2-4852-b725-a2f9ce6c13ce" (UID: "96fb560a-8ea2-4852-b725-a2f9ce6c13ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:22.041539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.041512 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.043240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.043213 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh" pod "96fb560a-8ea2-4852-b725-a2f9ce6c13ce" (UID: "96fb560a-8ea2-4852-b725-a2f9ce6c13ce"). InnerVolumeSpecName "kube-api-access-rzjnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:22.043351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.043316 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs" pod "96fb560a-8ea2-4852-b725-a2f9ce6c13ce" (UID: "96fb560a-8ea2-4852-b725-a2f9ce6c13ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:22.043538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.043518 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls" pod "96fb560a-8ea2-4852-b725-a2f9ce6c13ce" (UID: "96fb560a-8ea2-4852-b725-a2f9ce6c13ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:22.143567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.143532 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.143567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.143567 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.143836 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.143579 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjnh\" (UniqueName: \"kubernetes.io/projected/96fb560a-8ea2-4852-b725-a2f9ce6c13ce-kube-api-access-rzjnh\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.465217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.465188 5136 generic.go:356] "Generic (PLEG): container finished" podID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" containerID="dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992" exitCode=0 Nov 02 20:25:22.465382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.465233 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" Nov 02 20:25:22.465382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.465271 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" event={"ID":"96fb560a-8ea2-4852-b725-a2f9ce6c13ce","Type":"ContainerDied","Data":"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992"} Nov 02 20:25:22.465382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.465329 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8" event={"ID":"96fb560a-8ea2-4852-b725-a2f9ce6c13ce","Type":"ContainerDied","Data":"c008ef2dcdde2cdb494450d5e2d4f1e8713a670f142609c945545292b55cfefe"} Nov 02 20:25:22.465382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.465359 5136 scope.go:122] "RemoveContainer" containerID="dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992" Nov 02 20:25:22.476025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.476004 5136 scope.go:122] "RemoveContainer" containerID="dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992" Nov 02 20:25:22.476314 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:22.476296 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992\": container with ID starting with dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992 not found: ID does not exist" containerID="dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992" Nov 02 20:25:22.476370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.476322 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992"} err="failed to get container status \"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992\": rpc error: code = NotFound desc = could not find container \"dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992\": container with ID starting with dc1920bf3ce077ba683b338242892d85256fac8fffa1bc513ad6410b1f0b5992 not found: ID does not exist" Nov 02 20:25:22.620971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.620942 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:25:22.623895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:22.623868 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f83bc2-67b6cc46d6-tv7j8"] Nov 02 20:25:23.358869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.358832 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:25:23.359548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.359530 5136 cpu_manager.go:414] "containerMap: removing container" podUID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" containerName="maas-api" Nov 02 20:25:23.359592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.359550 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" containerName="maas-api" Nov 02 20:25:23.359656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.359647 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" containerName="maas-api" Nov 02 20:25:23.364491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.364471 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.421656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.421606 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:25:23.459983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.459948 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b028e6-serving-cert\"" Nov 02 20:25:23.463835 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.463805 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.463980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.463944 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.464197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.464174 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.464250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.464217 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz5s\" (UniqueName: \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.565293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.565264 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.565293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.565302 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz5s\" (UniqueName: \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.565556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.565384 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.565556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.565440 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.565556 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:23.565540 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-serving-cert: secret "maas-api-e2e-praxis-a-b028e6-serving-cert" not found Nov 02 20:25:23.565759 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:23.565610 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls podName:3fb2b3b5-e709-448d-bd5c-166455e4b288 nodeName:}" failed. No retries permitted until 2026-11-02 20:25:24.065594359 +0000 UTC m=+2207.672138849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls") pod "maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" (UID: "3fb2b3b5-e709-448d-bd5c-166455e4b288") : secret "maas-api-e2e-praxis-a-b028e6-serving-cert" not found Nov 02 20:25:23.565970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.565950 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.567943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.567923 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:23.625393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:23.625362 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz5s\" (UniqueName: \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:24.081752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:24.080567 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:24.090753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:24.083380 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:24.278776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:24.277702 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:24.547381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:24.547344 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fb560a-8ea2-4852-b725-a2f9ce6c13ce" path="/var/lib/kubelet/pods/96fb560a-8ea2-4852-b725-a2f9ce6c13ce/volumes" Nov 02 20:25:24.655894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:24.655864 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:25:24.657001 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:25:24.656977 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb2b3b5_e709_448d_bd5c_166455e4b288.slice/crio-e55572cb191163a3e4a91011353b4dd29fbed5785924942ca13e19f98e67f053 WatchSource:0}: Error finding container e55572cb191163a3e4a91011353b4dd29fbed5785924942ca13e19f98e67f053: Status 404 returned error can't find the container with id e55572cb191163a3e4a91011353b4dd29fbed5785924942ca13e19f98e67f053 Nov 02 20:25:25.486882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:25.486848 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" event={"ID":"3fb2b3b5-e709-448d-bd5c-166455e4b288","Type":"ContainerStarted","Data":"e55572cb191163a3e4a91011353b4dd29fbed5785924942ca13e19f98e67f053"} Nov 02 20:25:25.849705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:25.849618 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:25:25.850218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:25.849877 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" containerID="cri-o://2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c" gracePeriod=30 Nov 02 20:25:26.495528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:26.495496 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" event={"ID":"3fb2b3b5-e709-448d-bd5c-166455e4b288","Type":"ContainerStarted","Data":"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27"} Nov 02 20:25:26.495702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:26.495565 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:26.605844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:26.605795 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" podStartSLOduration=2.651519048 podStartE2EDuration="3.60578165s" totalImagesPullingTime="954.262602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:26.605028048 +0000 UTC m=+2210.211572558" watchObservedRunningTime="2026-11-02 20:25:26.60578165 +0000 UTC m=+2210.212326148" Nov 02 20:25:29.487474 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:29.487436 5136 patch_prober.go:29] interesting pod/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:25:29.487884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:29.487512 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:25:31.121276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.121221 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:25:31.166798 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.166765 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.166942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.166821 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.166942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.166850 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.166942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.166871 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.166942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.166927 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.167095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167013 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.167159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167113 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.167217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167179 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.167217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167190 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:31.167217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167202 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:31.167380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167249 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\") pod \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\" (UID: \"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee\") " Nov 02 20:25:31.167380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167363 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:31.167654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167622 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:31.167654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167636 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:31.167915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167895 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.167967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167921 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.167967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167936 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.167967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167951 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.168061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.167964 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.169230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.169203 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:31.169314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.169253 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "kube-api-access-zdvml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:31.169487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.169469 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:31.169528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.169483 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo" pod "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" (UID: "48c1f526-3290-4984-b3fb-7dcfc7b1d9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:31.269318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.269286 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.269318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.269320 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.269528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.269335 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.269528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.269351 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdvml\" (UniqueName: \"kubernetes.io/projected/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee-kube-api-access-zdvml\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:31.529491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.529457 5136 generic.go:356] "Generic (PLEG): container finished" podID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerID="2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c" exitCode=0 Nov 02 20:25:31.529675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.529513 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" Nov 02 20:25:31.529675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.529544 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerDied","Data":"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c"} Nov 02 20:25:31.529675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.529617 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf" event={"ID":"48c1f526-3290-4984-b3fb-7dcfc7b1d9ee","Type":"ContainerDied","Data":"48b1bd7c3ee3d5197ec0469c238d229672c2fbf359d0af446fc34570205c3689"} Nov 02 20:25:31.529675 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.529655 5136 scope.go:122] "RemoveContainer" containerID="2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c" Nov 02 20:25:31.540404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.540383 5136 scope.go:122] "RemoveContainer" containerID="7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754" Nov 02 20:25:31.549647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.549628 5136 scope.go:122] "RemoveContainer" containerID="2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c" Nov 02 20:25:31.549900 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:31.549883 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c\": container with ID starting with 2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c not found: ID does not exist" containerID="2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c" Nov 02 20:25:31.549961 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.549908 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c"} err="failed to get container status \"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c\": rpc error: code = NotFound desc = could not find container \"2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c\": container with ID starting with 2429679fd2284cb48a12842944040c22c5e1e435cd1b50549625e5949e07634c not found: ID does not exist" Nov 02 20:25:31.549961 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.549924 5136 scope.go:122] "RemoveContainer" containerID="7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754" Nov 02 20:25:31.550179 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:31.550157 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754\": container with ID starting with 7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754 not found: ID does not exist" containerID="7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754" Nov 02 20:25:31.550225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.550185 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754"} err="failed to get container status \"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754\": rpc error: code = NotFound desc = could not find container \"7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754\": container with ID starting with 7a26dd926585d482d7d720e0e26f951f022c35e057c050e9801e8d434d4ab754 not found: ID does not exist" Nov 02 20:25:31.686183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.686154 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:25:31.733517 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:31.733484 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f83bc2-openshift-default-59d4b87b79-x77nf"] Nov 02 20:25:32.507073 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:32.507044 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:25:32.546868 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:25:32.546834 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" path="/var/lib/kubelet/pods/48c1f526-3290-4984-b3fb-7dcfc7b1d9ee/volumes" Nov 02 20:25:51.057875 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:25:51.057834 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:26:00.587032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.587001 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:26:00.588448 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.588421 5136 cpu_manager.go:414] "containerMap: removing container" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:26:00.588448 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.588448 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:26:00.588640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.588629 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:26:00.591612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.591591 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.641510 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.641481 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6f1fd4-serving-cert\"" Nov 02 20:26:00.643229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.643187 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:26:00.698882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.698854 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.699056 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.698897 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.699056 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.699035 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.699224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.699206 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdvpm\" (UniqueName: \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.743719 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.743642 5136 generic.go:356] "Generic (PLEG): container finished" podID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerID="400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7" exitCode=0 Nov 02 20:26:00.743888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.743719 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerDied","Data":"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7"} Nov 02 20:26:00.744202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.744181 5136 scope.go:122] "RemoveContainer" containerID="400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7" Nov 02 20:26:00.745954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.745922 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:00.746083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.746021 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:00.746083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.746073 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:00.799989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.799961 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.800148 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.800009 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.800148 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.800075 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.800308 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:00.800290 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-serving-cert: secret "maas-api-e2e-ipp-b-6f1fd4-serving-cert" not found Nov 02 20:26:00.800370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.800312 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdvpm\" (UniqueName: \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.800370 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:00.800352 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls podName:fde9eee6-ac24-4ea0-badf-b3f04815b50b nodeName:}" failed. No retries permitted until 2026-11-02 20:26:01.300331892 +0000 UTC m=+2244.906876368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls") pod "maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" (UID: "fde9eee6-ac24-4ea0-badf-b3f04815b50b") : secret "maas-api-e2e-ipp-b-6f1fd4-serving-cert" not found Nov 02 20:26:00.800852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.800831 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.802327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.802304 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:00.906512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:00.903796 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdvpm\" (UniqueName: \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:01.308060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:01.308014 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:01.310398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:01.310377 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:01.564131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:01.564039 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:01.753116 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:01.753077 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerStarted","Data":"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271"} Nov 02 20:26:01.967109 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:26:01.967083 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde9eee6_ac24_4ea0_badf_b3f04815b50b.slice/crio-3643429ae649b36179f0d6c06bcd8072c0bd5bf52c987358b443f42a16f068de WatchSource:0}: Error finding container 3643429ae649b36179f0d6c06bcd8072c0bd5bf52c987358b443f42a16f068de: Status 404 returned error can't find the container with id 3643429ae649b36179f0d6c06bcd8072c0bd5bf52c987358b443f42a16f068de Nov 02 20:26:01.968156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:01.968127 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:26:02.552272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:02.552244 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:26:02.553348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:02.553325 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 20:26:02.553468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:02.553373 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 20:26:02.766025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:02.765985 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" event={"ID":"fde9eee6-ac24-4ea0-badf-b3f04815b50b","Type":"ContainerStarted","Data":"3643429ae649b36179f0d6c06bcd8072c0bd5bf52c987358b443f42a16f068de"} Nov 02 20:26:03.016814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.016775 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:26:03.552989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.552953 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 20:26:03.553165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.553024 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 20:26:03.781338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.781305 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" event={"ID":"fde9eee6-ac24-4ea0-badf-b3f04815b50b","Type":"ContainerStarted","Data":"4d98c8bbe7522bc2378e30271ae4ad76125df2ddeeae837200fc192ccc71724c"} Nov 02 20:26:03.781823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.781544 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:03.914815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:03.914387 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" podStartSLOduration=3.026258545 podStartE2EDuration="3.914364076s" totalImagesPullingTime="888.105531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:03.894387032 +0000 UTC m=+2247.500931530" watchObservedRunningTime="2026-11-02 20:26:03.914364076 +0000 UTC m=+2247.520908574" Nov 02 20:26:04.552536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:04.552503 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 20:26:04.552704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:04.552550 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 20:26:05.552432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:05.552397 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 20:26:05.552861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:05.552452 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 20:26:06.553093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:06.553061 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 20:26:06.553554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:06.553115 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 20:26:07.556825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:07.556795 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:26:07.557776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:07.557726 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:26:09.793162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:09.793129 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:26:14.638323 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:14.638282 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:26:14.638823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:14.638539 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" containerID="cri-o://54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400" gracePeriod=30 Nov 02 20:26:19.902830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.902770 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:26:19.917470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.917438 5136 generic.go:356] "Generic (PLEG): container finished" podID="9af8216c-b210-494f-a975-96c702c654f1" containerID="54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400" exitCode=0 Nov 02 20:26:19.917616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.917485 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" Nov 02 20:26:19.917616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.917519 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerDied","Data":"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400"} Nov 02 20:26:19.917616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.917573 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk" event={"ID":"9af8216c-b210-494f-a975-96c702c654f1","Type":"ContainerDied","Data":"192d9b4ddc3f34cfb9b541761ab48fbcdd68f1622c3bbf6ff04a9b325cc9fc28"} Nov 02 20:26:19.917616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.917597 5136 scope.go:122] "RemoveContainer" containerID="54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400" Nov 02 20:26:19.927226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.927208 5136 scope.go:122] "RemoveContainer" containerID="80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85" Nov 02 20:26:19.936246 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.936226 5136 scope.go:122] "RemoveContainer" containerID="54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400" Nov 02 20:26:19.936516 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:19.936498 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400\": container with ID starting with 54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400 not found: ID does not exist" containerID="54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400" Nov 02 20:26:19.936610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.936524 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400"} err="failed to get container status \"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400\": rpc error: code = NotFound desc = could not find container \"54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400\": container with ID starting with 54be1912d64cdf0da372641ae562bedf4bfdb4396ed4c4d74e63b7efcfc63400 not found: ID does not exist" Nov 02 20:26:19.936610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.936541 5136 scope.go:122] "RemoveContainer" containerID="80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85" Nov 02 20:26:19.936825 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:19.936804 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85\": container with ID starting with 80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85 not found: ID does not exist" containerID="80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85" Nov 02 20:26:19.936876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:19.936831 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85"} err="failed to get container status \"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85\": rpc error: code = NotFound desc = could not find container \"80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85\": container with ID starting with 80476d7b113ebf2a891205cb3904ec86707a3bd792f47fccbbb5da7f9deeeb85 not found: ID does not exist" Nov 02 20:26:20.012958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.012932 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.012974 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013007 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013035 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013167 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013197 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013260 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013297 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013325 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:20.013433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013348 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:20.013433 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013364 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\") pod \"9af8216c-b210-494f-a975-96c702c654f1\" (UID: \"9af8216c-b210-494f-a975-96c702c654f1\") " Nov 02 20:26:20.013654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013584 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:20.013654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013641 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:20.013804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013782 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:20.014020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.013989 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.014020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.014013 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.014170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.014026 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.014170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.014039 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af8216c-b210-494f-a975-96c702c654f1-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.014170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.014051 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.015154 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.015125 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:20.015595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.015571 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "kube-api-access-dhnlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:20.015677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.015654 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:20.015677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.015670 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token" pod "9af8216c-b210-494f-a975-96c702c654f1" (UID: "9af8216c-b210-494f-a975-96c702c654f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:20.115534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.115501 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af8216c-b210-494f-a975-96c702c654f1-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.115534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.115529 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af8216c-b210-494f-a975-96c702c654f1-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.115534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.115539 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhnlr\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-kube-api-access-dhnlr\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.115791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.115549 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af8216c-b210-494f-a975-96c702c654f1-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:20.334692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.334595 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:26:20.337515 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.337489 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b8db2-openshift-default-55d66c567-h82bk"] Nov 02 20:26:20.552250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:20.552142 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af8216c-b210-494f-a975-96c702c654f1" path="/var/lib/kubelet/pods/9af8216c-b210-494f-a975-96c702c654f1/volumes" Nov 02 20:26:21.003770 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:21.003722 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:26:31.894283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:31.894250 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:31.894699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:31.894305 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:32.055410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.055306 5136 generic.go:356] "Generic (PLEG): container finished" podID="d750cdea-5020-42e7-9760-993e101592cd" containerID="3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe" exitCode=0 Nov 02 20:26:32.055590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.055402 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerDied","Data":"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe"} Nov 02 20:26:32.056528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.056498 5136 scope.go:122] "RemoveContainer" containerID="3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe" Nov 02 20:26:32.058612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.058577 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:32.058754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.058667 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:32.058754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:32.058708 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:26:33.063238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:33.063204 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerStarted","Data":"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61"} Nov 02 20:26:33.972233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:33.972199 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:26:33.972431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:33.972394 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:33.972490 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:33.972451 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:34.972599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:34.972564 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:34.973124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:34.972618 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:35.972049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:35.972013 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:35.972240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:35.972081 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:36.972142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:36.972111 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:36.972526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:36.972160 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:37.971766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:37.971724 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:26:37.971982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:37.971786 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:26:38.976456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:38.976421 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:26:38.977025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:38.976780 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:26:38.977399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:26:38.977375 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:26:51.005011 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:26:51.004968 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:27:21.004139 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:27:21.004107 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:27:23.544912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:23.544884 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vzlpf_fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f/payload-processing/0.log" Nov 02 20:27:23.803182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:23.803113 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vzlpf_fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f/payload-processing/0.log" Nov 02 20:27:29.713435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:29.713403 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vzlpf_fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f/payload-processing/0.log" Nov 02 20:27:29.958850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:29.958820 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vzlpf_fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f/payload-processing/0.log" Nov 02 20:27:42.199290 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:42.199259 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:27:42.199706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:42.199468 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" containerID="cri-o://61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d" gracePeriod=30 Nov 02 20:27:47.453848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.453823 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:27:47.553765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.553665 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:27:47.554271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554242 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:27:47.554271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554260 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:27:47.554473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554289 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" Nov 02 20:27:47.554473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554298 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" Nov 02 20:27:47.554473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554322 5136 cpu_manager.go:414] "containerMap: removing container" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:27:47.554473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554329 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:27:47.554663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554506 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:27:47.554663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554522 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerName="istio-proxy" Nov 02 20:27:47.554663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.554534 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c1f526-3290-4984-b3fb-7dcfc7b1d9ee" containerName="istio-proxy" Nov 02 20:27:47.557932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.557914 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.579794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.579399 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:27:47.597227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597200 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597246 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597339 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597382 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597415 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597466 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597500 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:47.597573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597509 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:47.597573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597508 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597692 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.597987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.597951 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:47.598126 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.598078 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:47.598198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.598126 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:27:47.598345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.598326 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\") pod \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\" (UID: \"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51\") " Nov 02 20:27:47.599045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.599020 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.599156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.599048 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.599156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.599064 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.599156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.599078 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.599156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.599092 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.600957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.600909 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:47.600957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.600924 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:27:47.600957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.600918 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:47.601173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.600934 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz" pod "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" (UID: "2bf5f1b4-3105-4a64-b23c-23fe7bf88d51"). InnerVolumeSpecName "kube-api-access-rgndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:47.607296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.607277 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1bbd80-serving-cert\"" Nov 02 20:27:47.611852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.611827 5136 generic.go:356] "Generic (PLEG): container finished" podID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" containerID="61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d" exitCode=0 Nov 02 20:27:47.611948 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.611870 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" event={"ID":"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51","Type":"ContainerDied","Data":"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d"} Nov 02 20:27:47.611948 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.611905 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" Nov 02 20:27:47.611948 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.611922 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt" event={"ID":"2bf5f1b4-3105-4a64-b23c-23fe7bf88d51","Type":"ContainerDied","Data":"6e5ddb6c2ed796ce3746f369a92fcfa72218a88d07257aedb38a376c35bc4bba"} Nov 02 20:27:47.611948 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.611947 5136 scope.go:122] "RemoveContainer" containerID="61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d" Nov 02 20:27:47.639859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.639840 5136 scope.go:122] "RemoveContainer" containerID="61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d" Nov 02 20:27:47.640126 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:27:47.640100 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d\": container with ID starting with 61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d not found: ID does not exist" containerID="61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d" Nov 02 20:27:47.640184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.640144 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d"} err="failed to get container status \"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d\": rpc error: code = NotFound desc = could not find container \"61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d\": container with ID starting with 61b19e7991fd32557910a96987cf2fd293ec3fdfc2e0d9d42e859a9a4910039d not found: ID does not exist" Nov 02 20:27:47.700395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700366 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.700566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700403 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.700618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700572 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zxd\" (UniqueName: \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.700711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700684 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.700950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700934 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.701027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700955 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.701027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700968 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.701027 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.700985 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgndz\" (UniqueName: \"kubernetes.io/projected/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51-kube-api-access-rgndz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:47.806814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.806699 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zxd\" (UniqueName: \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.807009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.806858 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.807009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.806968 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.807009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.806999 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.809199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.807850 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.816339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.816309 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.817451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.817414 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.863406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.863377 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zxd\" (UniqueName: \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\") pod \"maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.871296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.871269 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:47.896251 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.896224 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:27:47.899954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:47.899931 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4wnzt"] Nov 02 20:27:48.241618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:48.241592 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:27:48.242408 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:27:48.242380 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5049df9a_90bb_46c3_a310_d96264ac0912.slice/crio-19888b6ad6dcc2d82cf4c409cb8470126bd84461c05183b862af1729bf5ba22a WatchSource:0}: Error finding container 19888b6ad6dcc2d82cf4c409cb8470126bd84461c05183b862af1729bf5ba22a: Status 404 returned error can't find the container with id 19888b6ad6dcc2d82cf4c409cb8470126bd84461c05183b862af1729bf5ba22a Nov 02 20:27:48.547038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:48.546956 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf5f1b4-3105-4a64-b23c-23fe7bf88d51" path="/var/lib/kubelet/pods/2bf5f1b4-3105-4a64-b23c-23fe7bf88d51/volumes" Nov 02 20:27:48.619563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:48.619532 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" event={"ID":"5049df9a-90bb-46c3-a310-d96264ac0912","Type":"ContainerStarted","Data":"19888b6ad6dcc2d82cf4c409cb8470126bd84461c05183b862af1729bf5ba22a"} Nov 02 20:27:49.643904 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:49.643812 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" event={"ID":"5049df9a-90bb-46c3-a310-d96264ac0912","Type":"ContainerStarted","Data":"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee"} Nov 02 20:27:49.644500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:49.644481 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:27:49.754340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:49.754280 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" podStartSLOduration=1.697756936 podStartE2EDuration="2.754262788s" totalImagesPullingTime="1.056505852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:49.753571504 +0000 UTC m=+2353.360116003" watchObservedRunningTime="2026-11-02 20:27:49.754262788 +0000 UTC m=+2353.360807286" Nov 02 20:27:51.004087 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:27:51.004048 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:27:56.661664 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:27:56.661636 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:28:09.050282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.050249 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:28:09.050855 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.050840 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:28:09.050923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.050855 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:28:09.050984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.050974 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af8216c-b210-494f-a975-96c702c654f1" containerName="istio-proxy" Nov 02 20:28:09.054500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.054481 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.151990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.151957 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cdbffd-serving-cert\"" Nov 02 20:28:09.154615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.154592 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:28:09.202527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.202489 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.202746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.202563 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.202746 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.202624 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.202894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.202723 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hxz\" (UniqueName: \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.304371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.304273 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.304647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.304622 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.304888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.304839 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.317700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.304885 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hxz\" (UniqueName: \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.327440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.327404 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.327759 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:28:09.327721 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-serving-cert: secret "maas-api-e2e-praxis-b-cdbffd-serving-cert" not found Nov 02 20:28:09.327926 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:28:09.327914 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls podName:9fca0041-dad8-4957-b103-869de80b010f nodeName:}" failed. No retries permitted until 2026-11-02 20:28:09.827891108 +0000 UTC m=+2373.434435586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls") pod "maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" (UID: "9fca0041-dad8-4957-b103-869de80b010f") : secret "maas-api-e2e-praxis-b-cdbffd-serving-cert" not found Nov 02 20:28:09.336279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.336247 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.382771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.382725 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hxz\" (UniqueName: \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.911340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.911308 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.913534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.913513 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:09.967678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:09.967651 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:10.333154 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:10.333127 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:28:10.334660 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:28:10.334633 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fca0041_dad8_4957_b103_869de80b010f.slice/crio-ecbef0b2f064a3a11bd6f18b3ebe4606cea90406770f4eb1e19574917ffbe9ff WatchSource:0}: Error finding container ecbef0b2f064a3a11bd6f18b3ebe4606cea90406770f4eb1e19574917ffbe9ff: Status 404 returned error can't find the container with id ecbef0b2f064a3a11bd6f18b3ebe4606cea90406770f4eb1e19574917ffbe9ff Nov 02 20:28:10.767203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:10.767167 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" event={"ID":"9fca0041-dad8-4957-b103-869de80b010f","Type":"ContainerStarted","Data":"ecbef0b2f064a3a11bd6f18b3ebe4606cea90406770f4eb1e19574917ffbe9ff"} Nov 02 20:28:11.774702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:11.774656 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" event={"ID":"9fca0041-dad8-4957-b103-869de80b010f","Type":"ContainerStarted","Data":"da4e24520883fb164fb6a250cd1347e0a915a6c465b8cb7415ac5f45ae475e5e"} Nov 02 20:28:11.775111 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:11.774741 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:11.884969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:11.884921 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" podStartSLOduration=2.06635249 podStartE2EDuration="2.884907976s" totalImagesPullingTime="818.555486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:11.883035621 +0000 UTC m=+2375.489580119" watchObservedRunningTime="2026-11-02 20:28:11.884907976 +0000 UTC m=+2375.491452468" Nov 02 20:28:13.678577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:13.678544 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:28:13.679049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:13.678761 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" podUID="02713b1b-f196-4964-81d2-b4b004cf3767" containerName="maas-api" containerID="cri-o://6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289" gracePeriod=30 Nov 02 20:28:14.012305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.012275 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:28:14.034213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034189 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") pod \"02713b1b-f196-4964-81d2-b4b004cf3767\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " Nov 02 20:28:14.034358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034343 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\" (UniqueName: \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\") pod \"02713b1b-f196-4964-81d2-b4b004cf3767\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " Nov 02 20:28:14.034402 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034368 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\") pod \"02713b1b-f196-4964-81d2-b4b004cf3767\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " Nov 02 20:28:14.034402 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034393 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\") pod \"02713b1b-f196-4964-81d2-b4b004cf3767\" (UID: \"02713b1b-f196-4964-81d2-b4b004cf3767\") " Nov 02 20:28:14.034796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034748 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca" pod "02713b1b-f196-4964-81d2-b4b004cf3767" (UID: "02713b1b-f196-4964-81d2-b4b004cf3767"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:14.034928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.034903 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02713b1b-f196-4964-81d2-b4b004cf3767-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.036239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.036213 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls" pod "02713b1b-f196-4964-81d2-b4b004cf3767" (UID: "02713b1b-f196-4964-81d2-b4b004cf3767"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:14.036425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.036408 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs" pod "02713b1b-f196-4964-81d2-b4b004cf3767" (UID: "02713b1b-f196-4964-81d2-b4b004cf3767"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:14.036490 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.036416 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4" pod "02713b1b-f196-4964-81d2-b4b004cf3767" (UID: "02713b1b-f196-4964-81d2-b4b004cf3767"). InnerVolumeSpecName "kube-api-access-jzst4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:14.140000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.137686 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzst4\" (UniqueName: \"kubernetes.io/projected/02713b1b-f196-4964-81d2-b4b004cf3767-kube-api-access-jzst4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.140000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.137797 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.140000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.137816 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02713b1b-f196-4964-81d2-b4b004cf3767-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.820803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.819090 5136 generic.go:356] "Generic (PLEG): container finished" podID="02713b1b-f196-4964-81d2-b4b004cf3767" containerID="6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289" exitCode=0 Nov 02 20:28:14.820803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.819204 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" event={"ID":"02713b1b-f196-4964-81d2-b4b004cf3767","Type":"ContainerDied","Data":"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289"} Nov 02 20:28:14.820803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.819259 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" event={"ID":"02713b1b-f196-4964-81d2-b4b004cf3767","Type":"ContainerDied","Data":"1e6c4cc7bdbc798c3ebafa17812ad97854a27b30038c6c4b45dc234ab4208c25"} Nov 02 20:28:14.820803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.819290 5136 scope.go:122] "RemoveContainer" containerID="6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289" Nov 02 20:28:14.820803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.819469 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59" Nov 02 20:28:14.871879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.871546 5136 scope.go:122] "RemoveContainer" containerID="6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289" Nov 02 20:28:14.872150 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:28:14.872027 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289\": container with ID starting with 6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289 not found: ID does not exist" containerID="6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289" Nov 02 20:28:14.872150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.872067 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289"} err="failed to get container status \"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289\": rpc error: code = NotFound desc = could not find container \"6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289\": container with ID starting with 6beee1438383f64404186d8e142439b319114a7df2381e78217e4037fa824289 not found: ID does not exist" Nov 02 20:28:14.978791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.978755 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:28:14.984638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:14.984615 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-57e2cf-6c9967c974-wjm59"] Nov 02 20:28:16.547402 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:16.547368 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02713b1b-f196-4964-81d2-b4b004cf3767" path="/var/lib/kubelet/pods/02713b1b-f196-4964-81d2-b4b004cf3767/volumes" Nov 02 20:28:17.786466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:17.786433 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:28:21.004141 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:28:21.004104 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:28:27.774772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:27.774712 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:28:27.775343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:27.774956 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" podUID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" containerName="maas-api" containerID="cri-o://4d98c8bbe7522bc2378e30271ae4ad76125df2ddeeae837200fc192ccc71724c" gracePeriod=30 Nov 02 20:28:27.911087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:27.911060 5136 generic.go:356] "Generic (PLEG): container finished" podID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" containerID="4d98c8bbe7522bc2378e30271ae4ad76125df2ddeeae837200fc192ccc71724c" exitCode=0 Nov 02 20:28:27.911325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:27.911135 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" event={"ID":"fde9eee6-ac24-4ea0-badf-b3f04815b50b","Type":"ContainerDied","Data":"4d98c8bbe7522bc2378e30271ae4ad76125df2ddeeae837200fc192ccc71724c"} Nov 02 20:28:28.076972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.076941 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:28:28.160854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.160823 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\") pod \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " Nov 02 20:28:28.161023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.160914 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\" (UniqueName: \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\") pod \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " Nov 02 20:28:28.161140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.161123 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") pod \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " Nov 02 20:28:28.161205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.161189 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\") pod \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\" (UID: \"fde9eee6-ac24-4ea0-badf-b3f04815b50b\") " Nov 02 20:28:28.161606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.161572 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca" pod "fde9eee6-ac24-4ea0-badf-b3f04815b50b" (UID: "fde9eee6-ac24-4ea0-badf-b3f04815b50b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:28.163110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.163084 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm" pod "fde9eee6-ac24-4ea0-badf-b3f04815b50b" (UID: "fde9eee6-ac24-4ea0-badf-b3f04815b50b"). InnerVolumeSpecName "kube-api-access-pdvpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:28.163229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.163136 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs" pod "fde9eee6-ac24-4ea0-badf-b3f04815b50b" (UID: "fde9eee6-ac24-4ea0-badf-b3f04815b50b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:28.163229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.163172 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls" pod "fde9eee6-ac24-4ea0-badf-b3f04815b50b" (UID: "fde9eee6-ac24-4ea0-badf-b3f04815b50b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:28.263105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.263077 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:28.263105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.263102 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde9eee6-ac24-4ea0-badf-b3f04815b50b-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:28.263105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.263111 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde9eee6-ac24-4ea0-badf-b3f04815b50b-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:28.263334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.263120 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdvpm\" (UniqueName: \"kubernetes.io/projected/fde9eee6-ac24-4ea0-badf-b3f04815b50b-kube-api-access-pdvpm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:28.927882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.922218 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" Nov 02 20:28:28.927882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.922248 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq" event={"ID":"fde9eee6-ac24-4ea0-badf-b3f04815b50b","Type":"ContainerDied","Data":"3643429ae649b36179f0d6c06bcd8072c0bd5bf52c987358b443f42a16f068de"} Nov 02 20:28:28.927882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:28.922310 5136 scope.go:122] "RemoveContainer" containerID="4d98c8bbe7522bc2378e30271ae4ad76125df2ddeeae837200fc192ccc71724c" Nov 02 20:28:29.078333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:29.078300 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:28:29.125753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:29.125714 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6f1fd4-84c6d7cf4d-knwwq"] Nov 02 20:28:30.547108 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:28:30.547077 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" path="/var/lib/kubelet/pods/fde9eee6-ac24-4ea0-badf-b3f04815b50b/volumes" Nov 02 20:28:51.003766 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:28:51.003713 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.821260 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822350 5136 cpu_manager.go:414] "containerMap: removing container" podUID="02713b1b-f196-4964-81d2-b4b004cf3767" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822371 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02713b1b-f196-4964-81d2-b4b004cf3767" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822385 5136 cpu_manager.go:414] "containerMap: removing container" podUID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822392 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822522 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="02713b1b-f196-4964-81d2-b4b004cf3767" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.822535 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde9eee6-ac24-4ea0-badf-b3f04815b50b" containerName="maas-api" Nov 02 20:29:03.831755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.827688 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:03.865526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.865493 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:29:03.878303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.878259 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:03.878507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.878313 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qf5\" (UniqueName: \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:03.909931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.909894 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fbe1eb\"" Nov 02 20:29:03.909931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.909910 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fbe1eb-dockercfg-r8zg8\"" Nov 02 20:29:03.979882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.979846 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:03.979882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.979884 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qf5\" (UniqueName: \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:03.980584 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:03.980562 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:04.034228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:04.034200 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qf5\" (UniqueName: \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\") pod \"payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:04.160051 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:04.159957 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:04.527474 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:04.527441 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:29:04.528025 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:29:04.527986 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc828216_c0f6_471b_bd43_0b6e6d9a63a0.slice/crio-55fcfa1f8eef56a8fc3f5e40c0009f5872b16dddf700a010b82c4207c0a0c609 WatchSource:0}: Error finding container 55fcfa1f8eef56a8fc3f5e40c0009f5872b16dddf700a010b82c4207c0a0c609: Status 404 returned error can't find the container with id 55fcfa1f8eef56a8fc3f5e40c0009f5872b16dddf700a010b82c4207c0a0c609 Nov 02 20:29:05.133799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:05.133764 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" event={"ID":"dc828216-c0f6-471b-bd43-0b6e6d9a63a0","Type":"ContainerStarted","Data":"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda"} Nov 02 20:29:05.134189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:05.133815 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" event={"ID":"dc828216-c0f6-471b-bd43-0b6e6d9a63a0","Type":"ContainerStarted","Data":"55fcfa1f8eef56a8fc3f5e40c0009f5872b16dddf700a010b82c4207c0a0c609"} Nov 02 20:29:05.134189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:05.133846 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:05.243857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:05.243810 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" podStartSLOduration=2.243797388 podStartE2EDuration="2.243797388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:05.24221645 +0000 UTC m=+2428.848760948" watchObservedRunningTime="2026-11-02 20:29:05.243797388 +0000 UTC m=+2428.850341886" Nov 02 20:29:16.144292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.144260 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:29:16.611693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.611653 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:29:16.611927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.611883 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" podUID="0944088f-62a5-429d-9f1d-ce06399029b5" containerName="maas-api" containerID="cri-o://17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7" gracePeriod=30 Nov 02 20:29:16.909422 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.909398 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:29:16.970753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.969789 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\") pod \"0944088f-62a5-429d-9f1d-ce06399029b5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " Nov 02 20:29:16.970753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.969883 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\") pod \"0944088f-62a5-429d-9f1d-ce06399029b5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " Nov 02 20:29:16.970753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.969938 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") pod \"0944088f-62a5-429d-9f1d-ce06399029b5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " Nov 02 20:29:16.970753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.970061 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\" (UniqueName: \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\") pod \"0944088f-62a5-429d-9f1d-ce06399029b5\" (UID: \"0944088f-62a5-429d-9f1d-ce06399029b5\") " Nov 02 20:29:16.974110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.973811 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t" pod "0944088f-62a5-429d-9f1d-ce06399029b5" (UID: "0944088f-62a5-429d-9f1d-ce06399029b5"). InnerVolumeSpecName "kube-api-access-2s75t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:16.976336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.976139 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs" pod "0944088f-62a5-429d-9f1d-ce06399029b5" (UID: "0944088f-62a5-429d-9f1d-ce06399029b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:16.976826 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.976556 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca" pod "0944088f-62a5-429d-9f1d-ce06399029b5" (UID: "0944088f-62a5-429d-9f1d-ce06399029b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:16.978676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:16.978649 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls" pod "0944088f-62a5-429d-9f1d-ce06399029b5" (UID: "0944088f-62a5-429d-9f1d-ce06399029b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:17.077926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.077878 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:17.078199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.078178 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0944088f-62a5-429d-9f1d-ce06399029b5-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:17.078321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.078307 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0944088f-62a5-429d-9f1d-ce06399029b5-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:17.078422 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.078407 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s75t\" (UniqueName: \"kubernetes.io/projected/0944088f-62a5-429d-9f1d-ce06399029b5-kube-api-access-2s75t\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:17.210444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.210398 5136 generic.go:356] "Generic (PLEG): container finished" podID="0944088f-62a5-429d-9f1d-ce06399029b5" containerID="17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7" exitCode=0 Nov 02 20:29:17.211067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.210498 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" Nov 02 20:29:17.211067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.210551 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" event={"ID":"0944088f-62a5-429d-9f1d-ce06399029b5","Type":"ContainerDied","Data":"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7"} Nov 02 20:29:17.211067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.210634 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5" event={"ID":"0944088f-62a5-429d-9f1d-ce06399029b5","Type":"ContainerDied","Data":"3ca428a7edc53924ea23f5790841ecd94341e6e47afe110b7ac6226e428c9cff"} Nov 02 20:29:17.211067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.210664 5136 scope.go:122] "RemoveContainer" containerID="17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7" Nov 02 20:29:17.222721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.222700 5136 scope.go:122] "RemoveContainer" containerID="17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7" Nov 02 20:29:17.223065 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:17.223023 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7\": container with ID starting with 17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7 not found: ID does not exist" containerID="17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7" Nov 02 20:29:17.223163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.223066 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7"} err="failed to get container status \"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7\": rpc error: code = NotFound desc = could not find container \"17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7\": container with ID starting with 17c4ca80db34de6aae5be45c58f11a80c9b03f2dfd88cdfc430ee6f5251989d7 not found: ID does not exist" Nov 02 20:29:17.368481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.368449 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:29:17.372177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:17.372152 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e87e9a-79d758c8ff-7xbd5"] Nov 02 20:29:18.546130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:18.546099 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0944088f-62a5-429d-9f1d-ce06399029b5" path="/var/lib/kubelet/pods/0944088f-62a5-429d-9f1d-ce06399029b5/volumes" Nov 02 20:29:19.495541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:19.495510 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:29:19.595365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:19.595332 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:29:20.549271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:20.549240 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:29:20.549499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:20.549473 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" containerID="cri-o://130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75" gracePeriod=30 Nov 02 20:29:21.012615 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:21.012566 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:29:24.496004 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.495963 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" podUID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" containerName="payload-processing" containerID="cri-o://695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793" gracePeriod=25 Nov 02 20:29:24.596338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.596282 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" podUID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" containerName="payload-pre-processing" containerID="cri-o://9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9" gracePeriod=25 Nov 02 20:29:24.774178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.774154 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:29:24.871145 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.871123 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:29:24.888069 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.888039 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\" (UniqueName: \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\") pod \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " Nov 02 20:29:24.888297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.888119 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\") pod \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\" (UID: \"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201\") " Nov 02 20:29:24.888863 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.888834 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume" pod "c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" (UID: "c646ffaf-d3bc-45a6-ae45-37b9c9ffd201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:24.891299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.891274 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b" pod "c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" (UID: "c646ffaf-d3bc-45a6-ae45-37b9c9ffd201"). InnerVolumeSpecName "kube-api-access-9jn7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:24.989951 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.989922 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\" (UniqueName: \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\") pod \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " Nov 02 20:29:24.990157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.990072 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\") pod \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\" (UID: \"73c99c37-52fe-4dbf-907b-5f29c6b19f10\") " Nov 02 20:29:24.990377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.990362 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn7b\" (UniqueName: \"kubernetes.io/projected/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-kube-api-access-9jn7b\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:24.990377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.990377 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:24.990467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.990404 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume" pod "73c99c37-52fe-4dbf-907b-5f29c6b19f10" (UID: "73c99c37-52fe-4dbf-907b-5f29c6b19f10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:24.992032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:24.992010 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr" pod "73c99c37-52fe-4dbf-907b-5f29c6b19f10" (UID: "73c99c37-52fe-4dbf-907b-5f29c6b19f10"). InnerVolumeSpecName "kube-api-access-7m8jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:25.091685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.091584 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8jr\" (UniqueName: \"kubernetes.io/projected/73c99c37-52fe-4dbf-907b-5f29c6b19f10-kube-api-access-7m8jr\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.091685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.091616 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c99c37-52fe-4dbf-907b-5f29c6b19f10-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.291224 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.291190 5136 generic.go:356] "Generic (PLEG): container finished" podID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" containerID="695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793" exitCode=0 Nov 02 20:29:25.291397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.291242 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" Nov 02 20:29:25.291397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.291277 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" event={"ID":"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201","Type":"ContainerDied","Data":"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793"} Nov 02 20:29:25.291397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.291330 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6" event={"ID":"c646ffaf-d3bc-45a6-ae45-37b9c9ffd201","Type":"ContainerDied","Data":"544b621b5cfb9063d51674b42c385e75495c802a907e464bf26559e7645d845f"} Nov 02 20:29:25.291397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.291360 5136 scope.go:122] "RemoveContainer" containerID="695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793" Nov 02 20:29:25.292743 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.292707 5136 generic.go:356] "Generic (PLEG): container finished" podID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" containerID="9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9" exitCode=0 Nov 02 20:29:25.292870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.292768 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" Nov 02 20:29:25.292870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.292771 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" event={"ID":"73c99c37-52fe-4dbf-907b-5f29c6b19f10","Type":"ContainerDied","Data":"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9"} Nov 02 20:29:25.292870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.292822 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm" event={"ID":"73c99c37-52fe-4dbf-907b-5f29c6b19f10","Type":"ContainerDied","Data":"1fe117685ba2ad048d5f6f7c770b0abdd533dcfb0c884ede6ddde4a00520c03e"} Nov 02 20:29:25.301370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.301353 5136 scope.go:122] "RemoveContainer" containerID="695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793" Nov 02 20:29:25.301631 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:25.301612 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793\": container with ID starting with 695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793 not found: ID does not exist" containerID="695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793" Nov 02 20:29:25.301753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.301637 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793"} err="failed to get container status \"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793\": rpc error: code = NotFound desc = could not find container \"695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793\": container with ID starting with 695a6bfa0039181c4e8a28471b6938442e47b2d8d1dd311e6d83cd0a9dee1793 not found: ID does not exist" Nov 02 20:29:25.301753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.301653 5136 scope.go:122] "RemoveContainer" containerID="9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9" Nov 02 20:29:25.310198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.310180 5136 scope.go:122] "RemoveContainer" containerID="9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9" Nov 02 20:29:25.310422 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:25.310406 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9\": container with ID starting with 9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9 not found: ID does not exist" containerID="9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9" Nov 02 20:29:25.310462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.310429 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9"} err="failed to get container status \"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9\": rpc error: code = NotFound desc = could not find container \"9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9\": container with ID starting with 9dc34bf7e447963e3e816a880d6d3337bcfb8601c928d4cf3f71348f64e345d9 not found: ID does not exist" Nov 02 20:29:25.447307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.447268 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:29:25.449932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.449908 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ed4910-7589b5cfff-n82pm"] Nov 02 20:29:25.552878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.552850 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:29:25.558229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.558205 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ed4910-7b5699f79c-8dlq6"] Nov 02 20:29:25.811883 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.811859 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:29:25.899971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.899943 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.899980 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900079 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900124 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900174 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900213 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900375 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900272 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900372 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.900493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900444 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900476 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:25.900598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900515 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\") pod \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\" (UID: \"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3\") " Nov 02 20:29:25.900668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900603 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.900794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900772 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.900876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.900863 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.901085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.901065 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.901166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.901089 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.901166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.901105 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.901166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.901121 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.901166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.901137 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.902449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.902423 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:25.902545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.902488 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.902599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.902545 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "kube-api-access-nhdtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:25.902599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:25.902568 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo" pod "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" (UID: "9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:26.002268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.002191 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:26.002268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.002216 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:26.002268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.002226 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:26.002268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.002234 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdtw\" (UniqueName: \"kubernetes.io/projected/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3-kube-api-access-nhdtw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:26.301383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.301289 5136 generic.go:356] "Generic (PLEG): container finished" podID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerID="130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75" exitCode=0 Nov 02 20:29:26.301383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.301359 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" Nov 02 20:29:26.301609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.301370 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" event={"ID":"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3","Type":"ContainerDied","Data":"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75"} Nov 02 20:29:26.301609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.301464 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt" event={"ID":"9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3","Type":"ContainerDied","Data":"5a8f6f7c03f326005bf9eeaa0b1e38f92934066a797b1ede09e4a6f2fc6be7fd"} Nov 02 20:29:26.301609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.301493 5136 scope.go:122] "RemoveContainer" containerID="130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75" Nov 02 20:29:26.311933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.311915 5136 scope.go:122] "RemoveContainer" containerID="130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75" Nov 02 20:29:26.312191 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:26.312170 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75\": container with ID starting with 130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75 not found: ID does not exist" containerID="130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75" Nov 02 20:29:26.312228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.312199 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75"} err="failed to get container status \"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75\": rpc error: code = NotFound desc = could not find container \"130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75\": container with ID starting with 130ecf2ebd0a3f33d043b5d05f009197a215e45158d044bca34f2bd032438a75 not found: ID does not exist" Nov 02 20:29:26.457811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.457774 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:29:26.505010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.504976 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7cjlt"] Nov 02 20:29:26.547988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.547955 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" path="/var/lib/kubelet/pods/73c99c37-52fe-4dbf-907b-5f29c6b19f10/volumes" Nov 02 20:29:26.548477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.548461 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" path="/var/lib/kubelet/pods/9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3/volumes" Nov 02 20:29:26.548839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:26.548827 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" path="/var/lib/kubelet/pods/c646ffaf-d3bc-45a6-ae45-37b9c9ffd201/volumes" Nov 02 20:29:51.003959 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:29:51.003920 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:29:51.318074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.317988 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:29:51.318263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.318237 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerName="maas-api" containerID="cri-o://02316dccbbe72c8cc711ede8fcfb5865b105a383f15a6895221d44a0aaef9c29" gracePeriod=30 Nov 02 20:29:51.395158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.395118 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.116:8443/health\": dial tcp 10.133.0.116:8443: connect: connection refused" Nov 02 20:29:51.454378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.454340 5136 generic.go:356] "Generic (PLEG): container finished" podID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerID="02316dccbbe72c8cc711ede8fcfb5865b105a383f15a6895221d44a0aaef9c29" exitCode=0 Nov 02 20:29:51.454533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.454411 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" event={"ID":"1280b5d6-7207-480f-a3d3-1da4974df00d","Type":"ContainerDied","Data":"02316dccbbe72c8cc711ede8fcfb5865b105a383f15a6895221d44a0aaef9c29"} Nov 02 20:29:51.606210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.606180 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:29:51.713044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713012 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\" (UniqueName: \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\") pod \"1280b5d6-7207-480f-a3d3-1da4974df00d\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " Nov 02 20:29:51.713188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713104 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\") pod \"1280b5d6-7207-480f-a3d3-1da4974df00d\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " Nov 02 20:29:51.713188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713175 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") pod \"1280b5d6-7207-480f-a3d3-1da4974df00d\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " Nov 02 20:29:51.713270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713208 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\") pod \"1280b5d6-7207-480f-a3d3-1da4974df00d\" (UID: \"1280b5d6-7207-480f-a3d3-1da4974df00d\") " Nov 02 20:29:51.713514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713479 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca" pod "1280b5d6-7207-480f-a3d3-1da4974df00d" (UID: "1280b5d6-7207-480f-a3d3-1da4974df00d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:51.713765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.713718 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1280b5d6-7207-480f-a3d3-1da4974df00d-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:51.716810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.716667 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs" pod "1280b5d6-7207-480f-a3d3-1da4974df00d" (UID: "1280b5d6-7207-480f-a3d3-1da4974df00d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:51.716810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.716795 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g" pod "1280b5d6-7207-480f-a3d3-1da4974df00d" (UID: "1280b5d6-7207-480f-a3d3-1da4974df00d"). InnerVolumeSpecName "kube-api-access-29g7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:51.716986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.716892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls" pod "1280b5d6-7207-480f-a3d3-1da4974df00d" (UID: "1280b5d6-7207-480f-a3d3-1da4974df00d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:51.814697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.814664 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29g7g\" (UniqueName: \"kubernetes.io/projected/1280b5d6-7207-480f-a3d3-1da4974df00d-kube-api-access-29g7g\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:51.814697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.814695 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:51.814697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:51.814704 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1280b5d6-7207-480f-a3d3-1da4974df00d-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:52.461672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:52.461645 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" Nov 02 20:29:52.462110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:52.461666 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd" event={"ID":"1280b5d6-7207-480f-a3d3-1da4974df00d","Type":"ContainerDied","Data":"366ae752470574ada039cf721e04d32b37bf2968e29d78aee9bd12a13038ba30"} Nov 02 20:29:52.462110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:52.461776 5136 scope.go:122] "RemoveContainer" containerID="02316dccbbe72c8cc711ede8fcfb5865b105a383f15a6895221d44a0aaef9c29" Nov 02 20:29:52.616611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:52.616574 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:29:52.664170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:52.664138 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ed4910-ddff7588f-9z2gd"] Nov 02 20:29:54.546679 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:54.546645 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" path="/var/lib/kubelet/pods/1280b5d6-7207-480f-a3d3-1da4974df00d/volumes" Nov 02 20:29:58.220297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:58.220263 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:29:58.220668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:29:58.220515 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" containerID="cri-o://f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61" gracePeriod=30 Nov 02 20:30:03.482106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.482076 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:30:03.531122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.531043 5136 generic.go:356] "Generic (PLEG): container finished" podID="d750cdea-5020-42e7-9760-993e101592cd" containerID="f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61" exitCode=0 Nov 02 20:30:03.531271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.531158 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" Nov 02 20:30:03.531271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.531179 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerDied","Data":"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61"} Nov 02 20:30:03.531271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.531235 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw" event={"ID":"d750cdea-5020-42e7-9760-993e101592cd","Type":"ContainerDied","Data":"433f3d59d0d7170a6bc0f45fabfe1e4002388ba7c9957eb1e5f1117da0d1bdc2"} Nov 02 20:30:03.531271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.531265 5136 scope.go:122] "RemoveContainer" containerID="f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61" Nov 02 20:30:03.540700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.540680 5136 scope.go:122] "RemoveContainer" containerID="3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe" Nov 02 20:30:03.548929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.548911 5136 scope.go:122] "RemoveContainer" containerID="f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61" Nov 02 20:30:03.549158 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:03.549140 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61\": container with ID starting with f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61 not found: ID does not exist" containerID="f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61" Nov 02 20:30:03.549207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.549167 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61"} err="failed to get container status \"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61\": rpc error: code = NotFound desc = could not find container \"f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61\": container with ID starting with f0b12407b0d1413ef4ca46d897ba2d8bbd1ec8cf44b986c607d013edeab4ef61 not found: ID does not exist" Nov 02 20:30:03.549207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.549183 5136 scope.go:122] "RemoveContainer" containerID="3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe" Nov 02 20:30:03.549395 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:03.549380 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe\": container with ID starting with 3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe not found: ID does not exist" containerID="3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe" Nov 02 20:30:03.549445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.549399 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe"} err="failed to get container status \"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe\": rpc error: code = NotFound desc = could not find container \"3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe\": container with ID starting with 3dabfdca8dbbd1bd3c7b8dd878c0f42deea47515c8036b2e5bf20faf7e8d3cfe not found: ID does not exist" Nov 02 20:30:03.553864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.553847 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.553934 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.553900 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.553985 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.553970 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554009 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554083 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554103 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:03.554129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554120 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554163 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554242 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554303 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:03.554395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554338 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\") pod \"d750cdea-5020-42e7-9760-993e101592cd\" (UID: \"d750cdea-5020-42e7-9760-993e101592cd\") " Nov 02 20:30:03.554566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554540 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:03.554653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554631 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:03.554837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554814 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:03.554983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554964 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.555052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.554989 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.555052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.555005 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.555052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.555018 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.555052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.555031 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750cdea-5020-42e7-9760-993e101592cd-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.556160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.556134 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:03.556558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.556532 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:03.556648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.556593 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:03.556694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.556646 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc" pod "d750cdea-5020-42e7-9760-993e101592cd" (UID: "d750cdea-5020-42e7-9760-993e101592cd"). InnerVolumeSpecName "kube-api-access-wzhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:03.656424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.656393 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750cdea-5020-42e7-9760-993e101592cd-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.656424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.656419 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhmc\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-kube-api-access-wzhmc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.656424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.656429 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750cdea-5020-42e7-9760-993e101592cd-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.656651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.656443 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750cdea-5020-42e7-9760-993e101592cd-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.946809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.946776 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:30:03.948984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:03.948956 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ed4910-openshift-default-78ddb5bc89-g7ggw"] Nov 02 20:30:04.546476 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:04.546436 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d750cdea-5020-42e7-9760-993e101592cd" path="/var/lib/kubelet/pods/d750cdea-5020-42e7-9760-993e101592cd/volumes" Nov 02 20:30:07.300514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.300477 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 20:30:07.300981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.300710 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" containerID="cri-o://418535d9375f2bb516f7119d93d6443159a8489197378f1b2d1414b2af62c4cf" gracePeriod=30 Nov 02 20:30:07.557388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.557360 5136 generic.go:356] "Generic (PLEG): container finished" podID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerID="418535d9375f2bb516f7119d93d6443159a8489197378f1b2d1414b2af62c4cf" exitCode=0 Nov 02 20:30:07.557524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.557435 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" event={"ID":"c8b7bbc7-a224-4e3b-bbf8-148479f943f4","Type":"ContainerDied","Data":"418535d9375f2bb516f7119d93d6443159a8489197378f1b2d1414b2af62c4cf"} Nov 02 20:30:07.586858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.586832 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 20:30:07.715165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715118 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.715365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715208 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.715365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715263 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.715365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715300 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.715365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715355 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.715612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715530 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.716665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715804 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\") pod \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\" (UID: \"c8b7bbc7-a224-4e3b-bbf8-148479f943f4\") " Nov 02 20:30:07.716665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.715891 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:07.716665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.716422 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-ca-configmap\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.718373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.718349 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:07.718626 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.718594 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:07.719028 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.719008 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:07.719125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.719044 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "kube-api-access-99k5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:07.720644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.720608 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:07.720929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.720904 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert" pod "c8b7bbc7-a224-4e3b-bbf8-148479f943f4" (UID: "c8b7bbc7-a224-4e3b-bbf8-148479f943f4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817538 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817562 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99k5q\" (UniqueName: \"kubernetes.io/projected/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-kube-api-access-99k5q\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817572 5136 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-local-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817581 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-csr-dns-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817589 5136 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-cacerts\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:07.817608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:07.817597 5136 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8b7bbc7-a224-4e3b-bbf8-148479f943f4-istio-kubeconfig\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:08.563892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.563868 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" Nov 02 20:30:08.563892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.563885 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62" event={"ID":"c8b7bbc7-a224-4e3b-bbf8-148479f943f4","Type":"ContainerDied","Data":"04358a6d77db0ca4e07ab845450823881cffafb2d31352cf73b35124c185ee07"} Nov 02 20:30:08.564337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.563934 5136 scope.go:122] "RemoveContainer" containerID="418535d9375f2bb516f7119d93d6443159a8489197378f1b2d1414b2af62c4cf" Nov 02 20:30:08.734560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.734532 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:30:08.735145 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735123 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" Nov 02 20:30:08.735145 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735145 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735155 5136 cpu_manager.go:414] "containerMap: removing container" podUID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" containerName="payload-pre-processing" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735161 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" containerName="payload-pre-processing" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735170 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0944088f-62a5-429d-9f1d-ce06399029b5" containerName="maas-api" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735174 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0944088f-62a5-429d-9f1d-ce06399029b5" containerName="maas-api" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735184 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735188 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735208 5136 cpu_manager.go:414] "containerMap: removing container" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735213 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735225 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerName="maas-api" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735230 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerName="maas-api" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735237 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" containerName="payload-processing" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735242 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" containerName="payload-processing" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735251 5136 cpu_manager.go:414] "containerMap: removing container" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735256 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735350 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735360 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1280b5d6-7207-480f-a3d3-1da4974df00d" containerName="maas-api" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735367 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c646ffaf-d3bc-45a6-ae45-37b9c9ffd201" containerName="payload-processing" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735374 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="d750cdea-5020-42e7-9760-993e101592cd" containerName="istio-proxy" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735383 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" containerName="discovery" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735388 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c99c37-52fe-4dbf-907b-5f29c6b19f10" containerName="payload-pre-processing" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735394 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0944088f-62a5-429d-9f1d-ce06399029b5" containerName="maas-api" Nov 02 20:30:08.735718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.735401 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6e9c49-fd36-4eb0-9a4c-05a78d6824d3" containerName="istio-proxy" Nov 02 20:30:08.740171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.740152 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:08.771181 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.771148 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 20:30:08.781781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.781757 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbh62"] Nov 02 20:30:08.832607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.832527 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:30:08.858040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.858006 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fbe1eb-serving-cert\"" Nov 02 20:30:08.929938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.929884 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:08.930100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.930029 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:08.930100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.930081 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:08.930204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:08.930129 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v522n\" (UniqueName: \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.030860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.030811 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.030860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.030866 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.031144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.030914 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v522n\" (UniqueName: \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.031144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.031023 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.031144 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:09.031116 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-serving-cert: secret "maas-api-e2e-praxis-b-fbe1eb-serving-cert" not found Nov 02 20:30:09.031317 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:09.031192 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls podName:16fb63ac-276c-4c4c-a27d-80b06ea0c97a nodeName:}" failed. No retries permitted until 2026-11-02 20:30:09.531172927 +0000 UTC m=+2493.137717403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls") pod "maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" (UID: "16fb63ac-276c-4c4c-a27d-80b06ea0c97a") : secret "maas-api-e2e-praxis-b-fbe1eb-serving-cert" not found Nov 02 20:30:09.031581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.031560 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.033423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.033402 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.085710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.085637 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v522n\" (UniqueName: \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.537670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.537630 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.540068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.540038 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:09.653860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:09.653819 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:10.048570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:10.048547 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:30:10.049669 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:30:10.049635 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fb63ac_276c_4c4c_a27d_80b06ea0c97a.slice/crio-6fb96213fa4ea3d0776cdf759e68796c34609a2315a870876f021c0ee33c0bc8 WatchSource:0}: Error finding container 6fb96213fa4ea3d0776cdf759e68796c34609a2315a870876f021c0ee33c0bc8: Status 404 returned error can't find the container with id 6fb96213fa4ea3d0776cdf759e68796c34609a2315a870876f021c0ee33c0bc8 Nov 02 20:30:10.051336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:10.051315 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:30:10.546532 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:10.546499 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b7bbc7-a224-4e3b-bbf8-148479f943f4" path="/var/lib/kubelet/pods/c8b7bbc7-a224-4e3b-bbf8-148479f943f4/volumes" Nov 02 20:30:10.576750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:10.576704 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" event={"ID":"16fb63ac-276c-4c4c-a27d-80b06ea0c97a","Type":"ContainerStarted","Data":"6fb96213fa4ea3d0776cdf759e68796c34609a2315a870876f021c0ee33c0bc8"} Nov 02 20:30:11.584127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:11.584081 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" event={"ID":"16fb63ac-276c-4c4c-a27d-80b06ea0c97a","Type":"ContainerStarted","Data":"017f71e4db6bb9ce03ba44bd76749917a3ffefb76e81e0e4681fc77d886f3324"} Nov 02 20:30:11.584634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:11.584221 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:11.697469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:11.697422 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" podStartSLOduration=2.8525231570000003 podStartE2EDuration="3.697408511s" totalImagesPullingTime="844.885354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:11.694258241 +0000 UTC m=+2495.300802740" watchObservedRunningTime="2026-11-02 20:30:11.697408511 +0000 UTC m=+2495.303953008" Nov 02 20:30:13.661544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:13.661512 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:30:13.662031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:13.661697 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" podUID="5049df9a-90bb-46c3-a310-d96264ac0912" containerName="maas-api" containerID="cri-o://79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee" gracePeriod=30 Nov 02 20:30:13.929798 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:13.929741 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:30:14.002375 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002343 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\") pod \"5049df9a-90bb-46c3-a310-d96264ac0912\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " Nov 02 20:30:14.002545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002505 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\" (UniqueName: \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\") pod \"5049df9a-90bb-46c3-a310-d96264ac0912\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " Nov 02 20:30:14.002612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002540 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\") pod \"5049df9a-90bb-46c3-a310-d96264ac0912\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " Nov 02 20:30:14.002668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002613 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\") pod \"5049df9a-90bb-46c3-a310-d96264ac0912\" (UID: \"5049df9a-90bb-46c3-a310-d96264ac0912\") " Nov 02 20:30:14.002725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002671 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca" pod "5049df9a-90bb-46c3-a310-d96264ac0912" (UID: "5049df9a-90bb-46c3-a310-d96264ac0912"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:14.002954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.002936 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5049df9a-90bb-46c3-a310-d96264ac0912-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.004657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.004632 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls" pod "5049df9a-90bb-46c3-a310-d96264ac0912" (UID: "5049df9a-90bb-46c3-a310-d96264ac0912"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:14.004657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.004648 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs" pod "5049df9a-90bb-46c3-a310-d96264ac0912" (UID: "5049df9a-90bb-46c3-a310-d96264ac0912"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:14.004788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.004693 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd" pod "5049df9a-90bb-46c3-a310-d96264ac0912" (UID: "5049df9a-90bb-46c3-a310-d96264ac0912"). InnerVolumeSpecName "kube-api-access-m8zxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:14.103917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.103883 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zxd\" (UniqueName: \"kubernetes.io/projected/5049df9a-90bb-46c3-a310-d96264ac0912-kube-api-access-m8zxd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.103917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.103913 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.104156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.103924 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5049df9a-90bb-46c3-a310-d96264ac0912-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.609350 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.609310 5136 generic.go:356] "Generic (PLEG): container finished" podID="5049df9a-90bb-46c3-a310-d96264ac0912" containerID="79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee" exitCode=0 Nov 02 20:30:14.609555 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.609377 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" event={"ID":"5049df9a-90bb-46c3-a310-d96264ac0912","Type":"ContainerDied","Data":"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee"} Nov 02 20:30:14.609555 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.609428 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" event={"ID":"5049df9a-90bb-46c3-a310-d96264ac0912","Type":"ContainerDied","Data":"19888b6ad6dcc2d82cf4c409cb8470126bd84461c05183b862af1729bf5ba22a"} Nov 02 20:30:14.609555 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.609457 5136 scope.go:122] "RemoveContainer" containerID="79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee" Nov 02 20:30:14.609774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.609589 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm" Nov 02 20:30:14.624590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.624563 5136 scope.go:122] "RemoveContainer" containerID="79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee" Nov 02 20:30:14.624996 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:14.624971 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee\": container with ID starting with 79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee not found: ID does not exist" containerID="79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee" Nov 02 20:30:14.625267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.625010 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee"} err="failed to get container status \"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee\": rpc error: code = NotFound desc = could not find container \"79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee\": container with ID starting with 79122c55206ea838996112777cd6122d97854d70f7cfa591cb891da878ba59ee not found: ID does not exist" Nov 02 20:30:14.765369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.765329 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:30:14.771047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.771022 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1bbd80-66978f7b4d-xnbmm"] Nov 02 20:30:14.824778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.824695 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:30:14.824964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:14.824918 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" podUID="3fb2b3b5-e709-448d-bd5c-166455e4b288" containerName="maas-api" containerID="cri-o://1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27" gracePeriod=30 Nov 02 20:30:15.088539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.088486 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:30:15.216017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.215984 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\") pod \"3fb2b3b5-e709-448d-bd5c-166455e4b288\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " Nov 02 20:30:15.216194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.216121 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") pod \"3fb2b3b5-e709-448d-bd5c-166455e4b288\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " Nov 02 20:30:15.216194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.216152 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\") pod \"3fb2b3b5-e709-448d-bd5c-166455e4b288\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " Nov 02 20:30:15.216329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.216199 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\" (UniqueName: \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\") pod \"3fb2b3b5-e709-448d-bd5c-166455e4b288\" (UID: \"3fb2b3b5-e709-448d-bd5c-166455e4b288\") " Nov 02 20:30:15.216394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.216376 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca" pod "3fb2b3b5-e709-448d-bd5c-166455e4b288" (UID: "3fb2b3b5-e709-448d-bd5c-166455e4b288"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:15.216849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.216826 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2b3b5-e709-448d-bd5c-166455e4b288-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.218532 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.218505 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs" pod "3fb2b3b5-e709-448d-bd5c-166455e4b288" (UID: "3fb2b3b5-e709-448d-bd5c-166455e4b288"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:15.218876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.218848 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls" pod "3fb2b3b5-e709-448d-bd5c-166455e4b288" (UID: "3fb2b3b5-e709-448d-bd5c-166455e4b288"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:15.219508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.219486 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s" pod "3fb2b3b5-e709-448d-bd5c-166455e4b288" (UID: "3fb2b3b5-e709-448d-bd5c-166455e4b288"). InnerVolumeSpecName "kube-api-access-cqz5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:15.318036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.318000 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.318036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.318027 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb2b3b5-e709-448d-bd5c-166455e4b288-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.318036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.318037 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz5s\" (UniqueName: \"kubernetes.io/projected/3fb2b3b5-e709-448d-bd5c-166455e4b288-kube-api-access-cqz5s\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.621133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.621102 5136 generic.go:356] "Generic (PLEG): container finished" podID="3fb2b3b5-e709-448d-bd5c-166455e4b288" containerID="1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27" exitCode=0 Nov 02 20:30:15.621308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.621148 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" Nov 02 20:30:15.621308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.621148 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" event={"ID":"3fb2b3b5-e709-448d-bd5c-166455e4b288","Type":"ContainerDied","Data":"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27"} Nov 02 20:30:15.621308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.621279 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc" event={"ID":"3fb2b3b5-e709-448d-bd5c-166455e4b288","Type":"ContainerDied","Data":"e55572cb191163a3e4a91011353b4dd29fbed5785924942ca13e19f98e67f053"} Nov 02 20:30:15.621495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.621310 5136 scope.go:122] "RemoveContainer" containerID="1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27" Nov 02 20:30:15.631142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.631127 5136 scope.go:122] "RemoveContainer" containerID="1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27" Nov 02 20:30:15.631356 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:15.631328 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27\": container with ID starting with 1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27 not found: ID does not exist" containerID="1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27" Nov 02 20:30:15.631456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.631360 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27"} err="failed to get container status \"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27\": rpc error: code = NotFound desc = could not find container \"1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27\": container with ID starting with 1179791d281527d61941da85cf3778a2045db964f6ee0e3c734761a4feb71e27 not found: ID does not exist" Nov 02 20:30:15.776263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.776234 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:30:15.782146 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:15.782117 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b028e6-5c4d4597d9-nh9xc"] Nov 02 20:30:16.547817 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:16.547788 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb2b3b5-e709-448d-bd5c-166455e4b288" path="/var/lib/kubelet/pods/3fb2b3b5-e709-448d-bd5c-166455e4b288/volumes" Nov 02 20:30:16.548231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:16.548216 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5049df9a-90bb-46c3-a310-d96264ac0912" path="/var/lib/kubelet/pods/5049df9a-90bb-46c3-a310-d96264ac0912/volumes" Nov 02 20:30:17.595102 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:17.595075 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:30:21.003868 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:21.003834 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:30:23.232361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:23.232330 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:30:23.232820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:23.232552 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" containerID="cri-o://6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271" gracePeriod=30 Nov 02 20:30:28.497811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.497752 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:30:28.597682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597655 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.597858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597702 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.597858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597770 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.597947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597865 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.597947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597900 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.598046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597947 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.598046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.597973 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.598046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598011 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.598201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598094 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\") pod \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\" (UID: \"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488\") " Nov 02 20:30:28.598201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598111 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:28.598384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598358 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:28.598470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598449 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:28.598585 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598559 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:28.598653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598567 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:28.598749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598717 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.598818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598752 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.598818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598765 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.598818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598774 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.598818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.598783 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.600053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.600030 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "kube-api-access-5mfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:28.600204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.600157 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:28.600365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.600346 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:28.600443 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.600428 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy" pod "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" (UID: "48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:28.695949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.695908 5136 generic.go:356] "Generic (PLEG): container finished" podID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerID="6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271" exitCode=0 Nov 02 20:30:28.696137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.695963 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" Nov 02 20:30:28.696137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.696000 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerDied","Data":"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271"} Nov 02 20:30:28.696137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.696077 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8" event={"ID":"48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488","Type":"ContainerDied","Data":"8f537c936a7eaa99b6be596e10e0723ea44c622ab4f654d967aeddc8accfab84"} Nov 02 20:30:28.696137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.696118 5136 scope.go:122] "RemoveContainer" containerID="6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271" Nov 02 20:30:28.699970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.699932 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.699970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.699962 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.700171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.699977 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.700171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.699993 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfr8\" (UniqueName: \"kubernetes.io/projected/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488-kube-api-access-5mfr8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.709644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.709614 5136 scope.go:122] "RemoveContainer" containerID="400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7" Nov 02 20:30:28.717972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.717955 5136 scope.go:122] "RemoveContainer" containerID="6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271" Nov 02 20:30:28.718218 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:28.718198 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271\": container with ID starting with 6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271 not found: ID does not exist" containerID="6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271" Nov 02 20:30:28.718295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.718231 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271"} err="failed to get container status \"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271\": rpc error: code = NotFound desc = could not find container \"6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271\": container with ID starting with 6ba8f9b5ed11da5ce1e023d32ed91dcc513858a23f4ad6b56bc90b7365ceb271 not found: ID does not exist" Nov 02 20:30:28.718295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.718257 5136 scope.go:122] "RemoveContainer" containerID="400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7" Nov 02 20:30:28.718510 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:28.718492 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7\": container with ID starting with 400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7 not found: ID does not exist" containerID="400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7" Nov 02 20:30:28.718550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.718518 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7"} err="failed to get container status \"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7\": rpc error: code = NotFound desc = could not find container \"400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7\": container with ID starting with 400e009af3ee984982479650ff13c8ea6d2a2cf13f4e2ac7cdff623edf1db8a7 not found: ID does not exist" Nov 02 20:30:28.853929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.853851 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:30:28.857811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:28.857785 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b028e6-openshift-default-5bcf55d777-wvnw8"] Nov 02 20:30:30.547843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:30.547802 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" path="/var/lib/kubelet/pods/48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488/volumes" Nov 02 20:30:32.497349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497316 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:30:32.497845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497830 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3fb2b3b5-e709-448d-bd5c-166455e4b288" containerName="maas-api" Nov 02 20:30:32.497897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497846 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb2b3b5-e709-448d-bd5c-166455e4b288" containerName="maas-api" Nov 02 20:30:32.497897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497872 5136 cpu_manager.go:414] "containerMap: removing container" podUID="5049df9a-90bb-46c3-a310-d96264ac0912" containerName="maas-api" Nov 02 20:30:32.497897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497877 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5049df9a-90bb-46c3-a310-d96264ac0912" containerName="maas-api" Nov 02 20:30:32.497897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497893 5136 cpu_manager.go:414] "containerMap: removing container" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497898 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497909 5136 cpu_manager.go:414] "containerMap: removing container" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497914 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.497994 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="5049df9a-90bb-46c3-a310-d96264ac0912" containerName="maas-api" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.498004 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.498012 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fa5caf-2a3a-4b54-9e2c-bd5d28cfd488" containerName="istio-proxy" Nov 02 20:30:32.498024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.498022 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb2b3b5-e709-448d-bd5c-166455e4b288" containerName="maas-api" Nov 02 20:30:32.501288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.501267 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557413 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557497 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557571 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557664 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557769 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557868 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557913 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557944 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.558754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.557997 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjxv\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.605924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.605892 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:30:32.609812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.609789 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b743f6-openshift-default-dockercfg-68tb2\"" Nov 02 20:30:32.659466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.659429 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.659672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.659481 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.659672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.659540 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.659974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.659953 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660066 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660013 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660090 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660209 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660243 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660288 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660315 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjxv\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660364 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660400 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.660763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.660714 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.661287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.661259 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.662813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.662788 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.664243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.664217 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.715554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.715523 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjxv\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.715699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.715531 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\") pod \"e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:32.814663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:32.814586 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:33.237952 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.237928 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:30:33.238820 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:30:33.238794 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37941455_7fe1_442e_aadd_e8a00eb3c975.slice/crio-0395054a68350c7c039fa4ca50bd07035a6dc7fdfe3e35e3e4958b4793220c43 WatchSource:0}: Error finding container 0395054a68350c7c039fa4ca50bd07035a6dc7fdfe3e35e3e4958b4793220c43: Status 404 returned error can't find the container with id 0395054a68350c7c039fa4ca50bd07035a6dc7fdfe3e35e3e4958b4793220c43 Nov 02 20:30:33.241403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.241366 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:33.241516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.241447 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:33.241516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.241497 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:33.736381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.736344 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerStarted","Data":"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f"} Nov 02 20:30:33.736792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.736412 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerStarted","Data":"0395054a68350c7c039fa4ca50bd07035a6dc7fdfe3e35e3e4958b4793220c43"} Nov 02 20:30:33.815019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.814987 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:33.846777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:33.846693 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podStartSLOduration=1.846677393 podStartE2EDuration="1.846677393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:33.84610838 +0000 UTC m=+2517.452652878" watchObservedRunningTime="2026-11-02 20:30:33.846677393 +0000 UTC m=+2517.453221892" Nov 02 20:30:34.822047 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:34.822021 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:35.747678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:35.747648 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:35.748606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:35.748581 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:30:40.177266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.177235 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:30:40.182691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.182673 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.233485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.233454 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:30:40.256124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.256078 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.261037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.261000 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt7x\" (UniqueName: \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.280081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.280045 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b743f6\"" Nov 02 20:30:40.280367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.280160 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b743f6-dockercfg-wwcw4\"" Nov 02 20:30:40.340595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.340566 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:30:40.344453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.344431 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.362659 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.362639 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.362765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.362723 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7x\" (UniqueName: \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.363367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.363347 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.397306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.397275 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:30:40.416360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.416330 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt7x\" (UniqueName: \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\") pod \"payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.463682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.463654 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.463853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.463808 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ljr\" (UniqueName: \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.496171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.496150 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:40.565467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.565438 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.565575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.565519 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ljr\" (UniqueName: \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.566429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.566409 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.619564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.619534 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ljr\" (UniqueName: \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\") pod \"payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.657956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.657931 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:40.861216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:40.861188 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:30:40.861958 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:30:40.861931 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca00e7e_1d9f_435a_b017_53e4fef0cd8b.slice/crio-05a521a04d37037392a859f7a0ed64684b48feac1b237df45636a351200a124b WatchSource:0}: Error finding container 05a521a04d37037392a859f7a0ed64684b48feac1b237df45636a351200a124b: Status 404 returned error can't find the container with id 05a521a04d37037392a859f7a0ed64684b48feac1b237df45636a351200a124b Nov 02 20:30:41.022916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.022894 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:30:41.024537 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:30:41.024512 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca63ae0_57c6_451c_8051_c3df1ea60256.slice/crio-83280878fbde227481d3d146ad8e48e7481f5240072637ecfb02e3052b297084 WatchSource:0}: Error finding container 83280878fbde227481d3d146ad8e48e7481f5240072637ecfb02e3052b297084: Status 404 returned error can't find the container with id 83280878fbde227481d3d146ad8e48e7481f5240072637ecfb02e3052b297084 Nov 02 20:30:41.781910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.781874 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" event={"ID":"9ca63ae0-57c6-451c-8051-c3df1ea60256","Type":"ContainerStarted","Data":"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad"} Nov 02 20:30:41.782371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.781926 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" event={"ID":"9ca63ae0-57c6-451c-8051-c3df1ea60256","Type":"ContainerStarted","Data":"83280878fbde227481d3d146ad8e48e7481f5240072637ecfb02e3052b297084"} Nov 02 20:30:41.782371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.781992 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:41.783588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.783564 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" event={"ID":"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b","Type":"ContainerStarted","Data":"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861"} Nov 02 20:30:41.783703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.783610 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:41.783703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.783627 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" event={"ID":"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b","Type":"ContainerStarted","Data":"05a521a04d37037392a859f7a0ed64684b48feac1b237df45636a351200a124b"} Nov 02 20:30:41.890711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:41.890665 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" podStartSLOduration=1.890651358 podStartE2EDuration="1.890651358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:41.889611489 +0000 UTC m=+2525.496155986" watchObservedRunningTime="2026-11-02 20:30:41.890651358 +0000 UTC m=+2525.497195875" Nov 02 20:30:42.000064 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:42.000016 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" podStartSLOduration=2.000000516 podStartE2EDuration="2.000000516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:41.997563623 +0000 UTC m=+2525.604108122" watchObservedRunningTime="2026-11-02 20:30:42.000000516 +0000 UTC m=+2525.606545034" Nov 02 20:30:51.004912 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:30:51.004828 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:30:52.792193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:52.792161 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:30:52.792579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:52.792230 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:30:55.349307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.349273 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:30:55.354907 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.354885 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.404924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.404895 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5c2879-openshift-default-dockercfg-46wpd\"" Nov 02 20:30:55.436919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.436891 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.436919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.436922 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.436948 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437040 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437085 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437145 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437210 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437283 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5s4\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.437351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.437338 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.454724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.454698 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538219 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5s4\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538287 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538355 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538379 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538417 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538456 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538507 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538545 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.538933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.538591 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.547369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.544162 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.547369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.544776 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.547369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.545014 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.547369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.545264 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.547369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.545931 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.555881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.551586 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.563668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.563635 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.598348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.598311 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5s4\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.602123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.602055 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\") pod \"e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:55.668928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:55.668886 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:56.086491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.086465 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:30:56.087527 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:30:56.087505 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce6098e_9322_459c_85ae_ea2f58b8fd5f.slice/crio-8947fbc9962221d17fedb7f7bbf627899e88ad4e98b50728b8d7ad6d8af59177 WatchSource:0}: Error finding container 8947fbc9962221d17fedb7f7bbf627899e88ad4e98b50728b8d7ad6d8af59177: Status 404 returned error can't find the container with id 8947fbc9962221d17fedb7f7bbf627899e88ad4e98b50728b8d7ad6d8af59177 Nov 02 20:30:56.090169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.090127 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:56.090259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.090197 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:56.090259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.090227 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:30:56.868515 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.868476 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerStarted","Data":"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8"} Nov 02 20:30:56.868919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:56.868526 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerStarted","Data":"8947fbc9962221d17fedb7f7bbf627899e88ad4e98b50728b8d7ad6d8af59177"} Nov 02 20:30:57.026812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:57.026762 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podStartSLOduration=2.026747551 podStartE2EDuration="2.026747551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:57.025062839 +0000 UTC m=+2540.631607349" watchObservedRunningTime="2026-11-02 20:30:57.026747551 +0000 UTC m=+2540.633292048" Nov 02 20:30:57.669075 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:57.669044 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:57.676112 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:57.676075 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:57.874162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:57.874134 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:30:57.875143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:30:57.875126 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:31:03.353546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.353510 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:31:03.357572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.357548 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.416321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.416262 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:31:03.454346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.454311 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5c2879\"" Nov 02 20:31:03.454521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.454311 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5c2879-dockercfg-ls5hx\"" Nov 02 20:31:03.516039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.516009 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:31:03.519823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.519803 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.536327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.536301 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.536457 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.536335 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.536505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.536454 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqk55\" (UniqueName: \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.536505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.536484 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ww7f\" (UniqueName: \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.572525 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.572485 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:31:03.637998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.637912 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.637998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.637946 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.638241 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.638026 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqk55\" (UniqueName: \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.638241 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.638044 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ww7f\" (UniqueName: \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.638800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.638772 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.638902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.638822 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.693360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.693324 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ww7f\" (UniqueName: \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\") pod \"payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.693523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.693497 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqk55\" (UniqueName: \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\") pod \"payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:03.832611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.832583 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:03.972136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:03.972098 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:04.202291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.202260 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:31:04.203179 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:31:04.203149 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd295d7a_1992_4bfe_b3bd_f4f36ee75cf6.slice/crio-ded245402881e1e27b3ab09be3ff8ff58f9a643ed217c109c3d7098b233136ec WatchSource:0}: Error finding container ded245402881e1e27b3ab09be3ff8ff58f9a643ed217c109c3d7098b233136ec: Status 404 returned error can't find the container with id ded245402881e1e27b3ab09be3ff8ff58f9a643ed217c109c3d7098b233136ec Nov 02 20:31:04.335933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.335881 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:31:04.336325 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:31:04.336301 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e84197d_2980_40cb_8646_7d19b48925d7.slice/crio-1e82e7e974f3a90b13601c0b6e0be7cebd77fbfa2d87129a92cbc451ff921c4a WatchSource:0}: Error finding container 1e82e7e974f3a90b13601c0b6e0be7cebd77fbfa2d87129a92cbc451ff921c4a: Status 404 returned error can't find the container with id 1e82e7e974f3a90b13601c0b6e0be7cebd77fbfa2d87129a92cbc451ff921c4a Nov 02 20:31:04.919534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.919498 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" event={"ID":"0e84197d-2980-40cb-8646-7d19b48925d7","Type":"ContainerStarted","Data":"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a"} Nov 02 20:31:04.920007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.919562 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" event={"ID":"0e84197d-2980-40cb-8646-7d19b48925d7","Type":"ContainerStarted","Data":"1e82e7e974f3a90b13601c0b6e0be7cebd77fbfa2d87129a92cbc451ff921c4a"} Nov 02 20:31:04.920007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.919692 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:04.921240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.921212 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" event={"ID":"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6","Type":"ContainerStarted","Data":"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1"} Nov 02 20:31:04.921370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.921262 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" event={"ID":"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6","Type":"ContainerStarted","Data":"ded245402881e1e27b3ab09be3ff8ff58f9a643ed217c109c3d7098b233136ec"} Nov 02 20:31:04.921370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:04.921301 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:05.031136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:05.031074 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" podStartSLOduration=2.031053673 podStartE2EDuration="2.031053673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:05.027963222 +0000 UTC m=+2548.634507722" watchObservedRunningTime="2026-11-02 20:31:05.031053673 +0000 UTC m=+2548.637598172" Nov 02 20:31:05.138583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:05.138529 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" podStartSLOduration=2.138514466 podStartE2EDuration="2.138514466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:05.136938611 +0000 UTC m=+2548.743483133" watchObservedRunningTime="2026-11-02 20:31:05.138514466 +0000 UTC m=+2548.745058963" Nov 02 20:31:06.331048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:06.331012 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:31:06.331488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:06.331213 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" containerID="cri-o://a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f" gracePeriod=30 Nov 02 20:31:11.073574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.073540 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:31:11.074084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.073597 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:31:11.589026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.589002 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:31:11.630460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630393 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630433 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630480 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630529 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630557 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630596 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.630705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630614 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.631035 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630775 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.631035 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630814 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\") pod \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\" (UID: \"108d74e8-b7d8-40ee-9f42-fde49744dc5f\") " Nov 02 20:31:11.631035 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630939 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:11.631200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.630977 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:11.631255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631205 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:11.631335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631309 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:11.631407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631356 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:11.631483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631461 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.631546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631490 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.631546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631507 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.631546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631523 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.631546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.631538 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.632795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.632756 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:11.632915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.632845 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:11.633085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.633063 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "kube-api-access-xd8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:11.633193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.633164 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo" pod "108d74e8-b7d8-40ee-9f42-fde49744dc5f" (UID: "108d74e8-b7d8-40ee-9f42-fde49744dc5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:11.732546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.732520 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8jh\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-kube-api-access-xd8jh\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.732546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.732543 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.732706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.732553 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.732706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.732562 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108d74e8-b7d8-40ee-9f42-fde49744dc5f-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.969006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.968967 5136 generic.go:356] "Generic (PLEG): container finished" podID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerID="a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f" exitCode=0 Nov 02 20:31:11.969197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.969028 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" Nov 02 20:31:11.969197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.969053 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" event={"ID":"108d74e8-b7d8-40ee-9f42-fde49744dc5f","Type":"ContainerDied","Data":"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f"} Nov 02 20:31:11.969197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.969110 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb" event={"ID":"108d74e8-b7d8-40ee-9f42-fde49744dc5f","Type":"ContainerDied","Data":"c6067c266e09133a4727f7c46fd3875a60df2ff90f46a803a052b8ba36330c51"} Nov 02 20:31:11.969197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.969136 5136 scope.go:122] "RemoveContainer" containerID="a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f" Nov 02 20:31:11.980196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.980180 5136 scope.go:122] "RemoveContainer" containerID="a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f" Nov 02 20:31:11.980446 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:11.980431 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f\": container with ID starting with a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f not found: ID does not exist" containerID="a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f" Nov 02 20:31:11.980521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:11.980452 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f"} err="failed to get container status \"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f\": rpc error: code = NotFound desc = could not find container \"a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f\": container with ID starting with a4868fb59b6d919f5cb43adc70a9ee826f135ee482257cc2b14ee7823bf3660f not found: ID does not exist" Nov 02 20:31:12.125714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:12.125683 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:31:12.173259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:12.173222 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cmhkb"] Nov 02 20:31:12.547023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:12.546992 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" path="/var/lib/kubelet/pods/108d74e8-b7d8-40ee-9f42-fde49744dc5f/volumes" Nov 02 20:31:15.799961 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:15.799929 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:31:15.800392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:15.800128 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" podUID="9fca0041-dad8-4957-b103-869de80b010f" containerName="maas-api" containerID="cri-o://da4e24520883fb164fb6a250cd1347e0a915a6c465b8cb7415ac5f45ae475e5e" gracePeriod=30 Nov 02 20:31:15.931080 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:15.931051 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:31:15.931529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:15.931500 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:31:16.007658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.007621 5136 generic.go:356] "Generic (PLEG): container finished" podID="9fca0041-dad8-4957-b103-869de80b010f" containerID="da4e24520883fb164fb6a250cd1347e0a915a6c465b8cb7415ac5f45ae475e5e" exitCode=0 Nov 02 20:31:16.007856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.007835 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" event={"ID":"9fca0041-dad8-4957-b103-869de80b010f","Type":"ContainerDied","Data":"da4e24520883fb164fb6a250cd1347e0a915a6c465b8cb7415ac5f45ae475e5e"} Nov 02 20:31:16.101033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.101005 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:31:16.178860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.178827 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\") pod \"9fca0041-dad8-4957-b103-869de80b010f\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " Nov 02 20:31:16.179032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.178956 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\") pod \"9fca0041-dad8-4957-b103-869de80b010f\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " Nov 02 20:31:16.179032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.179000 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\" (UniqueName: \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\") pod \"9fca0041-dad8-4957-b103-869de80b010f\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " Nov 02 20:31:16.179127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.179119 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") pod \"9fca0041-dad8-4957-b103-869de80b010f\" (UID: \"9fca0041-dad8-4957-b103-869de80b010f\") " Nov 02 20:31:16.179204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.179168 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca" pod "9fca0041-dad8-4957-b103-869de80b010f" (UID: "9fca0041-dad8-4957-b103-869de80b010f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:16.179570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.179546 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fca0041-dad8-4957-b103-869de80b010f-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:16.181158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.181127 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs" pod "9fca0041-dad8-4957-b103-869de80b010f" (UID: "9fca0041-dad8-4957-b103-869de80b010f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:16.181274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.181213 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz" pod "9fca0041-dad8-4957-b103-869de80b010f" (UID: "9fca0041-dad8-4957-b103-869de80b010f"). InnerVolumeSpecName "kube-api-access-88hxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:16.181586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.181568 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls" pod "9fca0041-dad8-4957-b103-869de80b010f" (UID: "9fca0041-dad8-4957-b103-869de80b010f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:16.280516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.280476 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:16.280516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.280509 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fca0041-dad8-4957-b103-869de80b010f-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:16.280792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:16.280526 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hxz\" (UniqueName: \"kubernetes.io/projected/9fca0041-dad8-4957-b103-869de80b010f-kube-api-access-88hxz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:17.015328 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:17.015295 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" Nov 02 20:31:17.015742 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:17.015300 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d" event={"ID":"9fca0041-dad8-4957-b103-869de80b010f","Type":"ContainerDied","Data":"ecbef0b2f064a3a11bd6f18b3ebe4606cea90406770f4eb1e19574917ffbe9ff"} Nov 02 20:31:17.015742 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:17.015429 5136 scope.go:122] "RemoveContainer" containerID="da4e24520883fb164fb6a250cd1347e0a915a6c465b8cb7415ac5f45ae475e5e" Nov 02 20:31:17.170216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:17.170187 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:31:17.217760 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:17.217714 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdbffd-7dcbb574d8-brv5d"] Nov 02 20:31:18.547077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:18.547047 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fca0041-dad8-4957-b103-869de80b010f" path="/var/lib/kubelet/pods/9fca0041-dad8-4957-b103-869de80b010f/volumes" Nov 02 20:31:21.004686 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:21.004646 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:31:28.197280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.197245 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:31:28.198060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198039 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9fca0041-dad8-4957-b103-869de80b010f" containerName="maas-api" Nov 02 20:31:28.198177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198061 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fca0041-dad8-4957-b103-869de80b010f" containerName="maas-api" Nov 02 20:31:28.198177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198075 5136 cpu_manager.go:414] "containerMap: removing container" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" Nov 02 20:31:28.198177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198083 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" Nov 02 20:31:28.198335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198219 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fca0041-dad8-4957-b103-869de80b010f" containerName="maas-api" Nov 02 20:31:28.198335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.198234 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="108d74e8-b7d8-40ee-9f42-fde49744dc5f" containerName="istio-proxy" Nov 02 20:31:28.203244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.203221 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.254194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.254163 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:31:28.298032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.298001 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b743f6-serving-cert\"" Nov 02 20:31:28.334997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.334966 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.335149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.335035 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.335149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.335121 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.335252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.335236 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6xw\" (UniqueName: \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.435984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.435952 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.436161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.436069 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6xw\" (UniqueName: \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.436161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.436100 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.436283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.436165 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.436283 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:28.436267 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-serving-cert: secret "maas-api-e2e-praxis-a-b743f6-serving-cert" not found Nov 02 20:31:28.436380 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:28.436323 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls podName:ce7be8d0-359f-4502-8f37-e593a2212291 nodeName:}" failed. No retries permitted until 2026-11-02 20:31:28.936305089 +0000 UTC m=+2572.542849565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls") pod "maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" (UID: "ce7be8d0-359f-4502-8f37-e593a2212291") : secret "maas-api-e2e-praxis-a-b743f6-serving-cert" not found Nov 02 20:31:28.436695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.436674 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.438467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.438446 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.490468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.490413 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6xw\" (UniqueName: \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.941855 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.941777 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:28.944190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:28.944168 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:29.117152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:29.117115 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:29.484921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:29.484893 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:31:29.485963 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:31:29.485931 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7be8d0_359f_4502_8f37_e593a2212291.slice/crio-828842e851c53b4b3547ab1c40cdf10b43ef3548267f0a8769f962870b1b0f26 WatchSource:0}: Error finding container 828842e851c53b4b3547ab1c40cdf10b43ef3548267f0a8769f962870b1b0f26: Status 404 returned error can't find the container with id 828842e851c53b4b3547ab1c40cdf10b43ef3548267f0a8769f962870b1b0f26 Nov 02 20:31:30.096928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:30.096895 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" event={"ID":"ce7be8d0-359f-4502-8f37-e593a2212291","Type":"ContainerStarted","Data":"828842e851c53b4b3547ab1c40cdf10b43ef3548267f0a8769f962870b1b0f26"} Nov 02 20:31:31.103833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:31.103789 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" event={"ID":"ce7be8d0-359f-4502-8f37-e593a2212291","Type":"ContainerStarted","Data":"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34"} Nov 02 20:31:31.104256 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:31.103980 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:31.212335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:31.212281 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" podStartSLOduration=2.317464979 podStartE2EDuration="3.212267468s" totalImagesPullingTime="894.802489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:31.211658568 +0000 UTC m=+2574.818203103" watchObservedRunningTime="2026-11-02 20:31:31.212267468 +0000 UTC m=+2574.818811973" Nov 02 20:31:37.116239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:37.116209 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:31:49.459211 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.459176 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:31:49.464538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.464514 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.516219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.516180 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:31:49.560212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.560172 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5c2879-serving-cert\"" Nov 02 20:31:49.574184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.574153 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.574322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.574274 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd89l\" (UniqueName: \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.574364 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.574329 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.574407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.574376 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.675774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.675721 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.675956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.675820 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.676011 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:49.675960 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-serving-cert: secret "maas-api-e2e-praxis-b-5c2879-serving-cert" not found Nov 02 20:31:49.676062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.676005 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.676129 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:49.676071 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls podName:7c3f3161-6a4e-430e-a579-89de94c921a8 nodeName:}" failed. No retries permitted until 2026-11-02 20:31:50.176043107 +0000 UTC m=+2593.782587603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls") pod "maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" (UID: "7c3f3161-6a4e-430e-a579-89de94c921a8") : secret "maas-api-e2e-praxis-b-5c2879-serving-cert" not found Nov 02 20:31:49.676307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.676283 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd89l\" (UniqueName: \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.676516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.676491 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.678572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.678553 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:49.731884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:49.731767 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd89l\" (UniqueName: \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:50.181964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:50.181886 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:50.184293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:50.184269 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:50.380090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:50.380043 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:50.750095 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:50.750070 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:31:50.752681 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:31:50.752652 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f3161_6a4e_430e_a579_89de94c921a8.slice/crio-b9000bb67646b19fe54cd470a17399fec36a073f14355d7a70972f0f192071fa WatchSource:0}: Error finding container b9000bb67646b19fe54cd470a17399fec36a073f14355d7a70972f0f192071fa: Status 404 returned error can't find the container with id b9000bb67646b19fe54cd470a17399fec36a073f14355d7a70972f0f192071fa Nov 02 20:31:51.012844 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:31:51.012751 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:31:51.230006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:51.229968 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" event={"ID":"7c3f3161-6a4e-430e-a579-89de94c921a8","Type":"ContainerStarted","Data":"b9000bb67646b19fe54cd470a17399fec36a073f14355d7a70972f0f192071fa"} Nov 02 20:31:52.236966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:52.236933 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" event={"ID":"7c3f3161-6a4e-430e-a579-89de94c921a8","Type":"ContainerStarted","Data":"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f"} Nov 02 20:31:52.237351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:52.236994 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:31:52.346799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:52.346719 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" podStartSLOduration=2.480514673 podStartE2EDuration="3.346698746s" totalImagesPullingTime="866.184073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:52.3455691 +0000 UTC m=+2595.952113599" watchObservedRunningTime="2026-11-02 20:31:52.346698746 +0000 UTC m=+2595.953243244" Nov 02 20:31:58.240298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:58.240263 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:31:58.247623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:31:58.247599 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:32:00.045481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.045447 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:32:00.045894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.045674 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" podUID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" containerName="maas-api" containerID="cri-o://017f71e4db6bb9ce03ba44bd76749917a3ffefb76e81e0e4681fc77d886f3324" gracePeriod=30 Nov 02 20:32:00.325196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.325095 5136 generic.go:356] "Generic (PLEG): container finished" podID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" containerID="017f71e4db6bb9ce03ba44bd76749917a3ffefb76e81e0e4681fc77d886f3324" exitCode=0 Nov 02 20:32:00.325373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.325209 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" event={"ID":"16fb63ac-276c-4c4c-a27d-80b06ea0c97a","Type":"ContainerDied","Data":"017f71e4db6bb9ce03ba44bd76749917a3ffefb76e81e0e4681fc77d886f3324"} Nov 02 20:32:00.370683 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.370657 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:32:00.456071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.456039 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\") pod \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " Nov 02 20:32:00.456349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.456123 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\" (UniqueName: \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\") pod \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " Nov 02 20:32:00.456589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.456564 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\") pod \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " Nov 02 20:32:00.456687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.456672 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") pod \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\" (UID: \"16fb63ac-276c-4c4c-a27d-80b06ea0c97a\") " Nov 02 20:32:00.456922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.456889 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca" pod "16fb63ac-276c-4c4c-a27d-80b06ea0c97a" (UID: "16fb63ac-276c-4c4c-a27d-80b06ea0c97a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:00.458338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.457274 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:00.460374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.459923 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs" pod "16fb63ac-276c-4c4c-a27d-80b06ea0c97a" (UID: "16fb63ac-276c-4c4c-a27d-80b06ea0c97a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:00.460374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.460031 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n" pod "16fb63ac-276c-4c4c-a27d-80b06ea0c97a" (UID: "16fb63ac-276c-4c4c-a27d-80b06ea0c97a"). InnerVolumeSpecName "kube-api-access-v522n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:00.476575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.463645 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls" pod "16fb63ac-276c-4c4c-a27d-80b06ea0c97a" (UID: "16fb63ac-276c-4c4c-a27d-80b06ea0c97a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:00.559487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.559462 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v522n\" (UniqueName: \"kubernetes.io/projected/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-kube-api-access-v522n\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:00.559487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.559486 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:00.559655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:00.559495 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16fb63ac-276c-4c4c-a27d-80b06ea0c97a-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.333094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:01.333064 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" Nov 02 20:32:01.333593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:01.333055 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp" event={"ID":"16fb63ac-276c-4c4c-a27d-80b06ea0c97a","Type":"ContainerDied","Data":"6fb96213fa4ea3d0776cdf759e68796c34609a2315a870876f021c0ee33c0bc8"} Nov 02 20:32:01.333593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:01.333205 5136 scope.go:122] "RemoveContainer" containerID="017f71e4db6bb9ce03ba44bd76749917a3ffefb76e81e0e4681fc77d886f3324" Nov 02 20:32:01.488974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:01.488944 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:32:01.491550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:01.491515 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fbe1eb-75dc9fb856-szmpp"] Nov 02 20:32:02.553100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:02.553056 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" path="/var/lib/kubelet/pods/16fb63ac-276c-4c4c-a27d-80b06ea0c97a/volumes" Nov 02 20:32:03.240827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.240769 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" podUID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" containerName="payload-processing" containerID="cri-o://af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda" gracePeriod=25 Nov 02 20:32:03.505933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.505867 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:32:03.599534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.599498 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\" (UniqueName: \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\") pod \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " Nov 02 20:32:03.599965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.599596 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\") pod \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\" (UID: \"dc828216-c0f6-471b-bd43-0b6e6d9a63a0\") " Nov 02 20:32:03.600023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.599985 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume" pod "dc828216-c0f6-471b-bd43-0b6e6d9a63a0" (UID: "dc828216-c0f6-471b-bd43-0b6e6d9a63a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:03.600278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.600255 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:03.601646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.601625 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5" pod "dc828216-c0f6-471b-bd43-0b6e6d9a63a0" (UID: "dc828216-c0f6-471b-bd43-0b6e6d9a63a0"). InnerVolumeSpecName "kube-api-access-z9qf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:03.701335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:03.701298 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qf5\" (UniqueName: \"kubernetes.io/projected/dc828216-c0f6-471b-bd43-0b6e6d9a63a0-kube-api-access-z9qf5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:04.354070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.354029 5136 generic.go:356] "Generic (PLEG): container finished" podID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" containerID="af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda" exitCode=0 Nov 02 20:32:04.354266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.354074 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" Nov 02 20:32:04.354266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.354109 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" event={"ID":"dc828216-c0f6-471b-bd43-0b6e6d9a63a0","Type":"ContainerDied","Data":"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda"} Nov 02 20:32:04.354266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.354159 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj" event={"ID":"dc828216-c0f6-471b-bd43-0b6e6d9a63a0","Type":"ContainerDied","Data":"55fcfa1f8eef56a8fc3f5e40c0009f5872b16dddf700a010b82c4207c0a0c609"} Nov 02 20:32:04.354266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.354181 5136 scope.go:122] "RemoveContainer" containerID="af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda" Nov 02 20:32:04.364426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.364409 5136 scope.go:122] "RemoveContainer" containerID="af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda" Nov 02 20:32:04.364686 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:32:04.364658 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda\": container with ID starting with af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda not found: ID does not exist" containerID="af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda" Nov 02 20:32:04.364809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.364694 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda"} err="failed to get container status \"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda\": rpc error: code = NotFound desc = could not find container \"af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda\": container with ID starting with af55efa75816bab95acc4c2629df2b8a5c727a308b22109846534dafac7a0cda not found: ID does not exist" Nov 02 20:32:04.510986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.510949 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:32:04.556754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:04.556715 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fbe1eb-6798497df9-ftxnj"] Nov 02 20:32:05.361183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.361108 5136 generic.go:356] "Generic (PLEG): container finished" podID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerID="777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f" exitCode=0 Nov 02 20:32:05.361660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.361180 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerDied","Data":"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f"} Nov 02 20:32:05.361660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.361591 5136 scope.go:122] "RemoveContainer" containerID="777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f" Nov 02 20:32:05.362973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.362947 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:05.363039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.363018 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:05.363076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.363050 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:05.748345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:05.748316 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:32:06.370485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:06.370446 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerStarted","Data":"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae"} Nov 02 20:32:06.546526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:06.546495 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" path="/var/lib/kubelet/pods/dc828216-c0f6-471b-bd43-0b6e6d9a63a0/volumes" Nov 02 20:32:06.818247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:06.815357 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:32:06.818247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:06.818103 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:32:06.818247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:06.818174 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 20:32:07.815690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:07.815648 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:32:07.816195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:07.815719 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 20:32:08.816005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:08.815972 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:32:08.816440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:08.816026 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 20:32:09.815548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:09.815511 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:32:09.815761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:09.815579 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 20:32:10.816760 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:10.816131 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:32:10.816760 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:10.816202 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 20:32:11.819360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:11.819329 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:32:11.819812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:11.819777 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:32:11.820398 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:11.820375 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:32:21.021952 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:32:21.021866 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:32:24.998215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998182 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:32:24.998825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998808 5136 cpu_manager.go:414] "containerMap: removing container" podUID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" containerName="payload-processing" Nov 02 20:32:24.998873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998828 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" containerName="payload-processing" Nov 02 20:32:24.998873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998842 5136 cpu_manager.go:414] "containerMap: removing container" podUID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" containerName="maas-api" Nov 02 20:32:24.998873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998847 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" containerName="maas-api" Nov 02 20:32:24.998976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998929 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fb63ac-276c-4c4c-a27d-80b06ea0c97a" containerName="maas-api" Nov 02 20:32:24.998976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:24.998947 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc828216-c0f6-471b-bd43-0b6e6d9a63a0" containerName="payload-processing" Nov 02 20:32:25.002295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.002276 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.073628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.073540 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.073893 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.073858 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.073904 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.073986 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.074105 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.074220 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.074269 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.074318 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhmd\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.074365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.074348 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.097583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.097550 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:32:25.141435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.141404 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-920071-openshift-default-dockercfg-7sbp7\"" Nov 02 20:32:25.175432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175399 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhmd\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175436 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175485 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175503 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175526 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175592 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175654 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175742 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.175973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.175790 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.176557 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.176505 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.177307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.177280 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.177427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.177313 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.177427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.177408 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.177558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.177478 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.178748 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.178705 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.178852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.178711 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.231658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.231591 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.235575 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.234994 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhmd\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\") pod \"e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.316513 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.316480 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:25.736407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.736368 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:32:25.773980 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:32:25.773950 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ced36e_bf60_4eb0_bcf7_0e4d6c9622ca.slice/crio-915dfc505a5ae619bb8b15189b4c4b9b15e5e99572a4ab4a912a004cf998db47 WatchSource:0}: Error finding container 915dfc505a5ae619bb8b15189b4c4b9b15e5e99572a4ab4a912a004cf998db47: Status 404 returned error can't find the container with id 915dfc505a5ae619bb8b15189b4c4b9b15e5e99572a4ab4a912a004cf998db47 Nov 02 20:32:25.776644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.776606 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:25.776769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.776682 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:25.776769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:25.776712 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:26.546523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:26.546491 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerStarted","Data":"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f"} Nov 02 20:32:26.547053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:26.546535 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerStarted","Data":"915dfc505a5ae619bb8b15189b4c4b9b15e5e99572a4ab4a912a004cf998db47"} Nov 02 20:32:26.657833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:26.656528 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podStartSLOduration=2.656510678 podStartE2EDuration="2.656510678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:26.652553063 +0000 UTC m=+2630.259097585" watchObservedRunningTime="2026-11-02 20:32:26.656510678 +0000 UTC m=+2630.263055179" Nov 02 20:32:27.216666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.216622 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:32:27.219963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.219942 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.269441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.269410 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-426941-openshift-default-dockercfg-mxlvf\"" Nov 02 20:32:27.302870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.302837 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.302922 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.302985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8qs\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303013 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303042 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303174 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303220 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303288 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.303392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.303380 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.317311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.317281 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:27.321064 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.321027 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:32:27.323674 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.323650 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:27.405113 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405075 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405164 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405199 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405266 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8qs\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405290 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405329 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405370 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405440 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405494 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405764 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405632 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.405914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405889 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.406103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.405940 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.406103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.406021 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.406277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.406255 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.407846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.407825 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.407941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.407876 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.462582 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.462545 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.462772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.462697 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8qs\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.532788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.532691 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:27.547406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.547378 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:27.548505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.548487 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:32:27.946558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.946526 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:32:27.946829 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:32:27.946806 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6fc5ad_1d5c_40ba_a339_3f0f2cc497c4.slice/crio-61fed7c8c0101f8b4385103ae39b758ec9eba3c3e151a2e4f16b5e6fb4bd010e WatchSource:0}: Error finding container 61fed7c8c0101f8b4385103ae39b758ec9eba3c3e151a2e4f16b5e6fb4bd010e: Status 404 returned error can't find the container with id 61fed7c8c0101f8b4385103ae39b758ec9eba3c3e151a2e4f16b5e6fb4bd010e Nov 02 20:32:27.949687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.949657 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:27.949782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.949722 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:27.949782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:27.949770 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:28.284023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.283779 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": read tcp 10.133.0.2:49174->10.133.0.126:15021: read: connection reset by peer" start-of-body= Nov 02 20:32:28.284023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.283863 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": read tcp 10.133.0.2:49174->10.133.0.126:15021: read: connection reset by peer" Nov 02 20:32:28.552905 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.552823 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" event={"ID":"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4","Type":"ContainerStarted","Data":"79ed3cbda72d4af8c94ebd087ca8ae1d2253644b90e09af75e6bfa1cb5abdc1b"} Nov 02 20:32:28.552905 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.552881 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" event={"ID":"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4","Type":"ContainerStarted","Data":"61fed7c8c0101f8b4385103ae39b758ec9eba3c3e151a2e4f16b5e6fb4bd010e"} Nov 02 20:32:28.554460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.554438 5136 generic.go:356] "Generic (PLEG): container finished" podID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerID="c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8" exitCode=0 Nov 02 20:32:28.554577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.554506 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerDied","Data":"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8"} Nov 02 20:32:28.554956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.554941 5136 scope.go:122] "RemoveContainer" containerID="c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8" Nov 02 20:32:28.556158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.556131 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:28.556242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.556196 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:28.556284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.556242 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:32:28.668427 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:28.668366 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podStartSLOduration=1.668346619 podStartE2EDuration="1.668346619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:28.666008747 +0000 UTC m=+2632.272553245" watchObservedRunningTime="2026-11-02 20:32:28.668346619 +0000 UTC m=+2632.274891117" Nov 02 20:32:29.537757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.534839 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:29.537757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.535323 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.537757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.535373 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:29.563782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.563741 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerStarted","Data":"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122"} Nov 02 20:32:29.669705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.669667 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:32:29.669912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.669889 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.669969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:29.669931 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:30.533754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:30.533106 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.533754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:30.533173 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:30.669751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:30.669696 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.670246 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:30.669797 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:31.536784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:31.533108 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.536784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:31.533169 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:31.669703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:31.669666 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.669929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:31.669752 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:32.534751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:32.534144 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:32.534751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:32.534206 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:32.669465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:32.669425 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:32.669638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:32.669492 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:33.154754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.154137 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:32:33.157755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.157720 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.197499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.197452 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5xm\" (UniqueName: \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.197885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.197864 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.262311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.251260 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-920071-dockercfg-4827s\"" Nov 02 20:32:33.264025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.264001 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-920071\"" Nov 02 20:32:33.289354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.289324 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:32:33.310776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.307592 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5xm\" (UniqueName: \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.310776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.307713 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.310776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.308686 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.384752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.384357 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5xm\" (UniqueName: \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\") pod \"payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.470143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.470114 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:33.549776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.548749 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:33.549776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.548795 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:33.669380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.669350 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:33.669559 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.669403 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:33.908777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:33.908675 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:32:34.533851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.533810 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:32:34.534252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.533878 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:32:34.617274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.617201 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" event={"ID":"06025fd0-f3ae-4e71-974d-fc6d41f36b4e","Type":"ContainerStarted","Data":"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64"} Nov 02 20:32:34.617691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.617663 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" event={"ID":"06025fd0-f3ae-4e71-974d-fc6d41f36b4e","Type":"ContainerStarted","Data":"0d001e3c491bb4c366b40407be89debd6c4e292a3265dd5418b2482ad0a3d9c4"} Nov 02 20:32:34.618466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.618441 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:34.669286 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.669252 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:32:34.669463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.669312 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:32:34.726769 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:34.726684 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" podStartSLOduration=1.726665139 podStartE2EDuration="1.726665139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:34.725882386 +0000 UTC m=+2638.332426886" watchObservedRunningTime="2026-11-02 20:32:34.726665139 +0000 UTC m=+2638.333209638" Nov 02 20:32:35.537454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.537427 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:35.537974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.537959 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:35.539838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.539811 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:32:35.673128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.673101 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:32:35.673492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.673469 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:32:35.674180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:35.674160 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:32:45.628263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:32:45.628234 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:32:51.003417 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:32:51.003382 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:33:15.258467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.258431 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:33:15.262374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.262352 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.287755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.286461 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.287755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.286548 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9zw\" (UniqueName: \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.287755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.286714 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.287755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.286762 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.311923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.311895 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-920071-serving-cert\"" Nov 02 20:33:15.361694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.361669 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:33:15.387957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.387925 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.388083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.387995 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9zw\" (UniqueName: \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.388153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.388137 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.388193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.388168 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.388370 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:15.388353 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-serving-cert: secret "maas-api-e2e-praxis-b-920071-serving-cert" not found Nov 02 20:33:15.388450 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:15.388439 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls podName:5f6ee98c-ac86-40c6-b281-1ee621cf2c9e nodeName:}" failed. No retries permitted until 2026-11-02 20:33:15.888418856 +0000 UTC m=+2679.494963335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls") pod "maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" (UID: "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e") : secret "maas-api-e2e-praxis-b-920071-serving-cert" not found Nov 02 20:33:15.388822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.388804 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.390413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.390392 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.441921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.441891 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9zw\" (UniqueName: \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.893685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.893654 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:15.896311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:15.896284 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:16.174721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:16.174642 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:16.548219 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:33:16.548182 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6ee98c_ac86_40c6_b281_1ee621cf2c9e.slice/crio-667620fee363d9b59ff64933a2816fa1a839b133a61eb66a90b97dff404f653b WatchSource:0}: Error finding container 667620fee363d9b59ff64933a2816fa1a839b133a61eb66a90b97dff404f653b: Status 404 returned error can't find the container with id 667620fee363d9b59ff64933a2816fa1a839b133a61eb66a90b97dff404f653b Nov 02 20:33:16.548988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:16.548968 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:33:16.872992 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:16.872915 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" event={"ID":"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e","Type":"ContainerStarted","Data":"667620fee363d9b59ff64933a2816fa1a839b133a61eb66a90b97dff404f653b"} Nov 02 20:33:17.881171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:17.881136 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" event={"ID":"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e","Type":"ContainerStarted","Data":"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1"} Nov 02 20:33:17.881590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:17.881269 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:17.990297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:17.990244 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" podStartSLOduration=2.145488771 podStartE2EDuration="2.990229014s" totalImagesPullingTime="844.740243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:17.98892075 +0000 UTC m=+2681.595465250" watchObservedRunningTime="2026-11-02 20:33:17.990229014 +0000 UTC m=+2681.596773512" Nov 02 20:33:21.003833 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:21.003800 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:33:23.893166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:23.893131 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:33:40.256847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:40.256814 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:33:40.356379 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:40.356346 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:33:45.258122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.258076 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" podUID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" containerName="payload-processing" containerID="cri-o://3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1" gracePeriod=25 Nov 02 20:33:45.356852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.356790 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" podUID="0e84197d-2980-40cb-8646-7d19b48925d7" containerName="payload-pre-processing" containerID="cri-o://7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a" gracePeriod=25 Nov 02 20:33:45.625788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.625765 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:33:45.651830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.651809 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:33:45.679789 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.679764 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\") pod \"0e84197d-2980-40cb-8646-7d19b48925d7\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " Nov 02 20:33:45.679890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.679803 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\" (UniqueName: \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\") pod \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " Nov 02 20:33:45.679932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.679908 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\" (UniqueName: \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\") pod \"0e84197d-2980-40cb-8646-7d19b48925d7\" (UID: \"0e84197d-2980-40cb-8646-7d19b48925d7\") " Nov 02 20:33:45.679972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.679955 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\") pod \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\" (UID: \"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6\") " Nov 02 20:33:45.680149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.680122 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume" pod "0e84197d-2980-40cb-8646-7d19b48925d7" (UID: "0e84197d-2980-40cb-8646-7d19b48925d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:45.680357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.680336 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume" pod "dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" (UID: "dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:45.680421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.680409 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e84197d-2980-40cb-8646-7d19b48925d7-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:45.681922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.681904 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f" pod "dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" (UID: "dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6"). InnerVolumeSpecName "kube-api-access-2ww7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:45.681980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.681931 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55" pod "0e84197d-2980-40cb-8646-7d19b48925d7" (UID: "0e84197d-2980-40cb-8646-7d19b48925d7"). InnerVolumeSpecName "kube-api-access-pqk55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:45.781950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.781879 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqk55\" (UniqueName: \"kubernetes.io/projected/0e84197d-2980-40cb-8646-7d19b48925d7-kube-api-access-pqk55\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:45.781950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.781905 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:45.781950 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:45.781915 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ww7f\" (UniqueName: \"kubernetes.io/projected/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6-kube-api-access-2ww7f\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:46.113867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.113787 5136 generic.go:356] "Generic (PLEG): container finished" podID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" containerID="3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1" exitCode=0 Nov 02 20:33:46.113867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.113836 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" Nov 02 20:33:46.114163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.113879 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" event={"ID":"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6","Type":"ContainerDied","Data":"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1"} Nov 02 20:33:46.114163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.113931 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss" event={"ID":"dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6","Type":"ContainerDied","Data":"ded245402881e1e27b3ab09be3ff8ff58f9a643ed217c109c3d7098b233136ec"} Nov 02 20:33:46.114163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.113958 5136 scope.go:122] "RemoveContainer" containerID="3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1" Nov 02 20:33:46.115366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.115343 5136 generic.go:356] "Generic (PLEG): container finished" podID="0e84197d-2980-40cb-8646-7d19b48925d7" containerID="7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a" exitCode=0 Nov 02 20:33:46.115488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.115369 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" event={"ID":"0e84197d-2980-40cb-8646-7d19b48925d7","Type":"ContainerDied","Data":"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a"} Nov 02 20:33:46.115488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.115384 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" Nov 02 20:33:46.115488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.115429 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4" event={"ID":"0e84197d-2980-40cb-8646-7d19b48925d7","Type":"ContainerDied","Data":"1e82e7e974f3a90b13601c0b6e0be7cebd77fbfa2d87129a92cbc451ff921c4a"} Nov 02 20:33:46.124062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.124041 5136 scope.go:122] "RemoveContainer" containerID="3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1" Nov 02 20:33:46.124380 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:46.124360 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1\": container with ID starting with 3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1 not found: ID does not exist" containerID="3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1" Nov 02 20:33:46.124472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.124393 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1"} err="failed to get container status \"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1\": rpc error: code = NotFound desc = could not find container \"3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1\": container with ID starting with 3ee6ff440c506fe9b169ae0df228970e3eac2513b65e564c73163e8e3a58bee1 not found: ID does not exist" Nov 02 20:33:46.124472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.124414 5136 scope.go:122] "RemoveContainer" containerID="7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a" Nov 02 20:33:46.132984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.132968 5136 scope.go:122] "RemoveContainer" containerID="7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a" Nov 02 20:33:46.133245 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:46.133226 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a\": container with ID starting with 7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a not found: ID does not exist" containerID="7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a" Nov 02 20:33:46.133333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.133249 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a"} err="failed to get container status \"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a\": rpc error: code = NotFound desc = could not find container \"7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a\": container with ID starting with 7dd714ede097b845a8d2525550b8060a113b146a9ad10b5e1f285aac801cac3a not found: ID does not exist" Nov 02 20:33:46.270219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.270190 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:33:46.317143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.317112 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5c2879-655cfbb97f-5g5q4"] Nov 02 20:33:46.376596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.376530 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:33:46.378854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.378832 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5c2879-7f8bd444d5-m4qss"] Nov 02 20:33:46.548706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.548616 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e84197d-2980-40cb-8646-7d19b48925d7" path="/var/lib/kubelet/pods/0e84197d-2980-40cb-8646-7d19b48925d7/volumes" Nov 02 20:33:46.549158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:46.549136 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" path="/var/lib/kubelet/pods/dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6/volumes" Nov 02 20:33:51.003251 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:33:51.003216 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:33:53.115753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.115046 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175612 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0e84197d-2980-40cb-8646-7d19b48925d7" containerName="payload-pre-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175646 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e84197d-2980-40cb-8646-7d19b48925d7" containerName="payload-pre-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175687 5136 cpu_manager.go:414] "containerMap: removing container" podUID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" containerName="payload-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175696 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" containerName="payload-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175909 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e84197d-2980-40cb-8646-7d19b48925d7" containerName="payload-pre-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.175930 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd295d7a-1992-4bfe-b3bd-f4f36ee75cf6" containerName="payload-processing" Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.181284 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 20:33:53.189753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.181401 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.333765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.331945 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2c6ce9\"" Nov 02 20:33:53.333765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.332097 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2c6ce9-dockercfg-pgbfr\"" Nov 02 20:33:53.333765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.333291 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.333765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.333336 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzwz\" (UniqueName: \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.461410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.459499 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.461410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.459557 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzwz\" (UniqueName: \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.461410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.461333 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.536318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.536210 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzwz\" (UniqueName: \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\") pod \"payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:53.724991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:53.724955 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:54.176779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:54.176752 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 20:33:54.178482 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:33:54.178447 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f98289b_28b8_4294_8e9c_2ea6b459c7cb.slice/crio-5e367fdfc261fc935e1fe6baef8c7f03c32ee6b1508f9bd8eed67cddd8c6a87b WatchSource:0}: Error finding container 5e367fdfc261fc935e1fe6baef8c7f03c32ee6b1508f9bd8eed67cddd8c6a87b: Status 404 returned error can't find the container with id 5e367fdfc261fc935e1fe6baef8c7f03c32ee6b1508f9bd8eed67cddd8c6a87b Nov 02 20:33:54.427499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:54.427403 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" event={"ID":"8f98289b-28b8-4294-8e9c-2ea6b459c7cb","Type":"ContainerStarted","Data":"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd"} Nov 02 20:33:54.427499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:54.427465 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" event={"ID":"8f98289b-28b8-4294-8e9c-2ea6b459c7cb","Type":"ContainerStarted","Data":"5e367fdfc261fc935e1fe6baef8c7f03c32ee6b1508f9bd8eed67cddd8c6a87b"} Nov 02 20:33:54.427703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:54.427603 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:33:54.536182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:54.536134 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" podStartSLOduration=1.536120526 podStartE2EDuration="1.536120526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:54.534696313 +0000 UTC m=+2718.141240810" watchObservedRunningTime="2026-11-02 20:33:54.536120526 +0000 UTC m=+2718.142665056" Nov 02 20:33:55.513782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.513721 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 20:33:55.518685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.518636 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.616672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.616641 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 20:33:55.621322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.621294 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.621723 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.621704 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.621959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.621941 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622126 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622110 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622274 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622413 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl5j\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622567 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622801 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.622992 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.622975 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.662259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.662202 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-44a484-openshift-default-dockercfg-qq7mk\"" Nov 02 20:33:55.724389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724351 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.724637 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724624 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.724722 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724713 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.724842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724832 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.724909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724899 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl5j\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.724975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.724967 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.725032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725023 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.725101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725075 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.725655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725087 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.725874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725856 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.726082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.726061 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.726082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725904 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.726227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725302 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.726227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.725539 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.726913 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.726895 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.727856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.727836 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.779652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.779576 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.779827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.779585 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl5j\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\") pod \"e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:55.864176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:55.864139 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:56.316489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.316456 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 20:33:56.318093 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:33:56.318065 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9619b5_758e_4ae0_a88d_d0ea4cd17988.slice/crio-629dc8ebc47d3edef41fbd5a0b8019146a81a05cfb9ce8dc87a807af57f983d8 WatchSource:0}: Error finding container 629dc8ebc47d3edef41fbd5a0b8019146a81a05cfb9ce8dc87a807af57f983d8: Status 404 returned error can't find the container with id 629dc8ebc47d3edef41fbd5a0b8019146a81a05cfb9ce8dc87a807af57f983d8 Nov 02 20:33:56.320636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.320606 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:33:56.320702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.320666 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:33:56.320702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.320695 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:33:56.444409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.444376 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerStarted","Data":"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843"} Nov 02 20:33:56.444637 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.444444 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerStarted","Data":"629dc8ebc47d3edef41fbd5a0b8019146a81a05cfb9ce8dc87a807af57f983d8"} Nov 02 20:33:56.558454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.558406 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podStartSLOduration=1.558393836 podStartE2EDuration="1.558393836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:56.556683347 +0000 UTC m=+2720.163227846" watchObservedRunningTime="2026-11-02 20:33:56.558393836 +0000 UTC m=+2720.164938334" Nov 02 20:33:56.864784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:56.864748 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:57.870156 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:57.870124 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:58.457071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:58.457036 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:33:58.457895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:33:58.457869 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:34:02.485873 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.485841 5136 generic.go:356] "Generic (PLEG): container finished" podID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerID="fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f" exitCode=0 Nov 02 20:34:02.486318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.485876 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerDied","Data":"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f"} Nov 02 20:34:02.486318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.486314 5136 scope.go:122] "RemoveContainer" containerID="fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f" Nov 02 20:34:02.487807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.487780 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:02.487872 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.487850 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:02.487907 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:02.487883 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:03.500917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:03.500803 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerStarted","Data":"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c"} Nov 02 20:34:04.317133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:04.317083 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:34:04.317330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:04.317313 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:34:04.317393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:04.317374 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 20:34:05.317251 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:05.317214 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:34:05.317662 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:05.317266 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 20:34:05.437033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:05.437005 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 20:34:06.317913 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:06.317876 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:34:06.318322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:06.317966 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 20:34:07.317090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:07.317048 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:34:07.317305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:07.317114 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 20:34:08.317416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:08.317373 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:34:08.317949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:08.317428 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 20:34:09.321327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:09.321299 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:34:09.321834 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:09.321789 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:34:09.322438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:09.322420 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:34:12.244076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.244038 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:34:12.248396 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.248371 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340069 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xfc\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340199 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340226 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340246 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340265 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340290 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340328 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340357 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.340589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.340391 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.347837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.347812 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:34:12.441684 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441657 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.441874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441757 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xfc\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.441874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441826 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.441874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441853 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.441874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441874 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441894 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.441975 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442060 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442217 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442254 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442267 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442268 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442479 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.442691 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.442661 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.444440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.444415 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.444898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.444880 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.496261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.496188 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.496261 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.496204 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xfc\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.561495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.561468 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:12.846604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.846527 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:34:12.846796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.846755 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" podUID="7c3f3161-6a4e-430e-a579-89de94c921a8" containerName="maas-api" containerID="cri-o://62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f" gracePeriod=30 Nov 02 20:34:12.983704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.983677 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:34:12.984302 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:12.984267 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453911ab_1871_4618_832a_48cfdd10aa20.slice/crio-d4ac08fa9b6c89df521a1a9b3418467c923b67759cd87d0931234b56cf2eb55e WatchSource:0}: Error finding container d4ac08fa9b6c89df521a1a9b3418467c923b67759cd87d0931234b56cf2eb55e: Status 404 returned error can't find the container with id d4ac08fa9b6c89df521a1a9b3418467c923b67759cd87d0931234b56cf2eb55e Nov 02 20:34:12.987487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.987449 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:12.987590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.987534 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:12.987590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:12.987578 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:13.130222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.130194 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:34:13.254536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.252865 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\" (UniqueName: \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\") pod \"7c3f3161-6a4e-430e-a579-89de94c921a8\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " Nov 02 20:34:13.254536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.253101 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\") pod \"7c3f3161-6a4e-430e-a579-89de94c921a8\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " Nov 02 20:34:13.254536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.253257 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\") pod \"7c3f3161-6a4e-430e-a579-89de94c921a8\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " Nov 02 20:34:13.254536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.253312 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") pod \"7c3f3161-6a4e-430e-a579-89de94c921a8\" (UID: \"7c3f3161-6a4e-430e-a579-89de94c921a8\") " Nov 02 20:34:13.256428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.255794 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca" pod "7c3f3161-6a4e-430e-a579-89de94c921a8" (UID: "7c3f3161-6a4e-430e-a579-89de94c921a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:13.257958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.257924 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls" pod "7c3f3161-6a4e-430e-a579-89de94c921a8" (UID: "7c3f3161-6a4e-430e-a579-89de94c921a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:13.260815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.260776 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs" pod "7c3f3161-6a4e-430e-a579-89de94c921a8" (UID: "7c3f3161-6a4e-430e-a579-89de94c921a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:13.261387 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.261354 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l" pod "7c3f3161-6a4e-430e-a579-89de94c921a8" (UID: "7c3f3161-6a4e-430e-a579-89de94c921a8"). InnerVolumeSpecName "kube-api-access-nd89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:13.354818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.354715 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:13.354818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.354773 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd89l\" (UniqueName: \"kubernetes.io/projected/7c3f3161-6a4e-430e-a579-89de94c921a8-kube-api-access-nd89l\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:13.354818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.354789 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3f3161-6a4e-430e-a579-89de94c921a8-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:13.354818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.354798 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f3161-6a4e-430e-a579-89de94c921a8-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:13.571550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.571516 5136 generic.go:356] "Generic (PLEG): container finished" podID="7c3f3161-6a4e-430e-a579-89de94c921a8" containerID="62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f" exitCode=0 Nov 02 20:34:13.571745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.571558 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" Nov 02 20:34:13.571745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.571609 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" event={"ID":"7c3f3161-6a4e-430e-a579-89de94c921a8","Type":"ContainerDied","Data":"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f"} Nov 02 20:34:13.571745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.571658 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx" event={"ID":"7c3f3161-6a4e-430e-a579-89de94c921a8","Type":"ContainerDied","Data":"b9000bb67646b19fe54cd470a17399fec36a073f14355d7a70972f0f192071fa"} Nov 02 20:34:13.571745 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.571685 5136 scope.go:122] "RemoveContainer" containerID="62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f" Nov 02 20:34:13.573501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.573476 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" event={"ID":"453911ab-1871-4618-832a-48cfdd10aa20","Type":"ContainerStarted","Data":"b159b455b923f6732a041fcabc78d1f1ee4d1af4b8815431985170520331e23f"} Nov 02 20:34:13.573622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.573529 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" event={"ID":"453911ab-1871-4618-832a-48cfdd10aa20","Type":"ContainerStarted","Data":"d4ac08fa9b6c89df521a1a9b3418467c923b67759cd87d0931234b56cf2eb55e"} Nov 02 20:34:13.581595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.581578 5136 scope.go:122] "RemoveContainer" containerID="62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f" Nov 02 20:34:13.581911 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:13.581892 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f\": container with ID starting with 62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f not found: ID does not exist" containerID="62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f" Nov 02 20:34:13.581975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.581919 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f"} err="failed to get container status \"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f\": rpc error: code = NotFound desc = could not find container \"62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f\": container with ID starting with 62b9f84c8d0126e39d33ea73e99cc2d8bf7bc93f73152681af6d4b6eb34b0d9f not found: ID does not exist" Nov 02 20:34:13.727670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.727618 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:34:13.731252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.731228 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5c2879-5bcd9b9c95-w9ntx"] Nov 02 20:34:13.838520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:13.838464 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podStartSLOduration=1.838448583 podStartE2EDuration="1.838448583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:13.837146603 +0000 UTC m=+2737.443691104" watchObservedRunningTime="2026-11-02 20:34:13.838448583 +0000 UTC m=+2737.444993080" Nov 02 20:34:14.013812 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.012313 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:34:14.014335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.014292 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7c3f3161-6a4e-430e-a579-89de94c921a8" containerName="maas-api" Nov 02 20:34:14.014479 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.014326 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3f3161-6a4e-430e-a579-89de94c921a8" containerName="maas-api" Nov 02 20:34:14.014599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.014583 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3f3161-6a4e-430e-a579-89de94c921a8" containerName="maas-api" Nov 02 20:34:14.019723 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.019697 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.060641 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.060611 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:34:14.069749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.069714 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b743f6\"/\"openshift-service-ca.crt\"" Nov 02 20:34:14.069902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.069787 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b743f6\"/\"kube-root-ca.crt\"" Nov 02 20:34:14.069902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.069856 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b743f6\"/\"praxis-route-b743f6-kserve-self-signed-certs\"" Nov 02 20:34:14.069902 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.069898 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b743f6\"/\"default-dockercfg-64466\"" Nov 02 20:34:14.182768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.182696 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.182768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.182765 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.183085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.183055 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9l8\" (UniqueName: \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.183170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.183152 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.183275 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.183255 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.183347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.183309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.284434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284340 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.284434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284392 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.284434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284421 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284442 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284600 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9l8\" (UniqueName: \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284677 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284812 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284855 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.285121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.284970 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.286829 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.286805 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.287189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.287162 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.338852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.338821 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9l8\" (UniqueName: \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\") pod \"praxis-route-b743f6-kserve-7478b6768f-58qqm\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:14.546899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.546823 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3f3161-6a4e-430e-a579-89de94c921a8" path="/var/lib/kubelet/pods/7c3f3161-6a4e-430e-a579-89de94c921a8/volumes" Nov 02 20:34:14.562573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.562548 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:14.564550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.564523 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:14.564666 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.564568 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:14.631604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:14.631570 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:15.052618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.052582 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:34:15.053839 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:15.053810 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0536ef81_e3b4_4e74_8904_3aa71db8f9ad.slice/crio-2e0b31c249e0e5f232a540f04a93fe0ab3b77b7716d95d87ff8c2fdf85bf6094 WatchSource:0}: Error finding container 2e0b31c249e0e5f232a540f04a93fe0ab3b77b7716d95d87ff8c2fdf85bf6094: Status 404 returned error can't find the container with id 2e0b31c249e0e5f232a540f04a93fe0ab3b77b7716d95d87ff8c2fdf85bf6094 Nov 02 20:34:15.562545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.562510 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:15.562991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.562559 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:15.587509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.587476 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" event={"ID":"0536ef81-e3b4-4e74-8904-3aa71db8f9ad","Type":"ContainerStarted","Data":"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad"} Nov 02 20:34:15.587627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.587521 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" event={"ID":"0536ef81-e3b4-4e74-8904-3aa71db8f9ad","Type":"ContainerStarted","Data":"2e0b31c249e0e5f232a540f04a93fe0ab3b77b7716d95d87ff8c2fdf85bf6094"} Nov 02 20:34:15.632599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.632571 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:15.711752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:15.701249 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" podStartSLOduration=2.701230675 podStartE2EDuration="2.701230675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:15.699086141 +0000 UTC m=+2739.305630642" watchObservedRunningTime="2026-11-02 20:34:15.701230675 +0000 UTC m=+2739.307775174" Nov 02 20:34:16.562203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:16.562170 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:16.562376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:16.562228 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:17.562056 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:17.562014 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:17.562601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:17.562062 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:18.567759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:18.566159 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:18.567759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:18.566214 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:19.563754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:19.562250 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 20:34:19.563754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:19.562309 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 20:34:20.035846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.035806 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:34:20.036370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.036071 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" containerID="cri-o://26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122" gracePeriod=30 Nov 02 20:34:20.622053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.622018 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:34:20.626657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.626633 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.676200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.676167 5136 patch_prober.go:29] interesting pod/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:34:20.676359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.676228 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:34:20.715271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.715240 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745341 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745431 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745469 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745531 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745631 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745670 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745748 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjcd\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745798 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.746461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.745843 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847133 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847096 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847144 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847166 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847220 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847591 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847704 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847681 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847688 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847832 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847933 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847906 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjcd\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847951 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847955 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.847993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.847981 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.848228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.848208 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.848475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.848446 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.850366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.850335 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.850533 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.850512 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.882493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.882421 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:20.882874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.882852 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:20.883793 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.883771 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:34:20.901530 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.901501 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjcd\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.902068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.902045 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:20.940927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:20.940900 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:21.024004 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:21.023968 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:34:21.357489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.357457 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:34:21.357866 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:21.357686 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2239908_f444_408c_801a_ccb0d1bc39ed.slice/crio-754e40789ab7afbc2c7419d5f3d0d65d5825e0667e25f31ca1351ab225c71eed WatchSource:0}: Error finding container 754e40789ab7afbc2c7419d5f3d0d65d5825e0667e25f31ca1351ab225c71eed: Status 404 returned error can't find the container with id 754e40789ab7afbc2c7419d5f3d0d65d5825e0667e25f31ca1351ab225c71eed Nov 02 20:34:21.360132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.360088 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:21.360233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.360160 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:21.360233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.360189 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:21.658219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.658123 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" event={"ID":"c2239908-f444-408c-801a-ccb0d1bc39ed","Type":"ContainerStarted","Data":"c7c7f894a20dcb12fbc6e5a0b7254f56b8ac2a332da6fea977a800060a0e8151"} Nov 02 20:34:21.658219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.658187 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" event={"ID":"c2239908-f444-408c-801a-ccb0d1bc39ed","Type":"ContainerStarted","Data":"754e40789ab7afbc2c7419d5f3d0d65d5825e0667e25f31ca1351ab225c71eed"} Nov 02 20:34:21.772824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.772764 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podStartSLOduration=1.7727167719999999 podStartE2EDuration="1.772716772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:21.771169781 +0000 UTC m=+2745.377714280" watchObservedRunningTime="2026-11-02 20:34:21.772716772 +0000 UTC m=+2745.379261271" Nov 02 20:34:21.941136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:21.941053 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:22.943663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:22.943628 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:22.944080 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:22.943678 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:23.941340 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:23.941309 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:23.941526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:23.941357 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:24.631896 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:24.631860 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:24.644705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:24.644673 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:24.714100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:24.714071 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:34:24.941592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:24.941496 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:24.941592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:24.941557 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:25.351226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.351201 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:34:25.533366 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533295 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533411 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533500 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533537 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533584 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533611 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533614 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533647 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533698 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.533779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.533763 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\") pod \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\" (UID: \"8ce6098e-9322-459c-85ae-ea2f58b8fd5f\") " Nov 02 20:34:25.534288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.534267 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.534400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.534353 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:25.534470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.534427 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:25.534529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.534460 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:25.534529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.534490 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:25.536795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.536764 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:34:25.536894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.536794 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:25.536894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.536801 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:25.536964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.536882 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4" pod "8ce6098e-9322-459c-85ae-ea2f58b8fd5f" (UID: "8ce6098e-9322-459c-85ae-ea2f58b8fd5f"). InnerVolumeSpecName "kube-api-access-lc5s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:25.635274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635228 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635372 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635408 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635437 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5s4\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-kube-api-access-lc5s4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635454 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635471 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635487 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.635699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.635502 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce6098e-9322-459c-85ae-ea2f58b8fd5f-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:25.685661 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.685631 5136 generic.go:356] "Generic (PLEG): container finished" podID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerID="26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122" exitCode=0 Nov 02 20:34:25.685881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.685682 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" Nov 02 20:34:25.685881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.685716 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerDied","Data":"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122"} Nov 02 20:34:25.685881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.685789 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d" event={"ID":"8ce6098e-9322-459c-85ae-ea2f58b8fd5f","Type":"ContainerDied","Data":"8947fbc9962221d17fedb7f7bbf627899e88ad4e98b50728b8d7ad6d8af59177"} Nov 02 20:34:25.685881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.685819 5136 scope.go:122] "RemoveContainer" containerID="26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122" Nov 02 20:34:25.721652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.721628 5136 scope.go:122] "RemoveContainer" containerID="c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8" Nov 02 20:34:25.733574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.733520 5136 scope.go:122] "RemoveContainer" containerID="26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122" Nov 02 20:34:25.733871 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:25.733842 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122\": container with ID starting with 26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122 not found: ID does not exist" containerID="26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122" Nov 02 20:34:25.733946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.733886 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122"} err="failed to get container status \"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122\": rpc error: code = NotFound desc = could not find container \"26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122\": container with ID starting with 26a5e3c539fb643fe88f65c968830fff8ac88ff29cfdff8be4e520ddcb7da122 not found: ID does not exist" Nov 02 20:34:25.733946 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.733908 5136 scope.go:122] "RemoveContainer" containerID="c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8" Nov 02 20:34:25.734188 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:25.734170 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8\": container with ID starting with c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8 not found: ID does not exist" containerID="c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8" Nov 02 20:34:25.734243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.734194 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8"} err="failed to get container status \"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8\": rpc error: code = NotFound desc = could not find container \"c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8\": container with ID starting with c9635210b88b795afff21c305a8453facb14581545f6f7d9b011611887c2ffc8 not found: ID does not exist" Nov 02 20:34:25.842478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.842405 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:34:25.855210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.855183 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5c2879-openshift-default-597d54b957-zfs4d"] Nov 02 20:34:25.942058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.942027 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:25.942263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:25.942085 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:26.547815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:26.547787 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" path="/var/lib/kubelet/pods/8ce6098e-9322-459c-85ae-ea2f58b8fd5f/volumes" Nov 02 20:34:26.941357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:26.941277 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:26.941357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:26.941332 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:27.942028 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:27.941993 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 20:34:27.942425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:27.942069 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 20:34:28.150467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.150424 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 20:34:28.151141 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.151119 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:28.151141 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.151142 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:28.151348 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.151334 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:28.151407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.151359 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:28.156524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.156503 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.164934 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.164911 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.164948 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lbk\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165049 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.164968 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165161 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165207 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165322 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165368 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165480 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.165637 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.165617 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.212486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.212454 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 20:34:28.266849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.266821 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.266849 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.266855 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lbk\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.266874 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.266916 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.266958 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267007 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267045 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267084 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.267618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267547 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.271756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267169 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.271756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267854 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.271756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.267932 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.271756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.268191 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.275752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.274944 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a9ed60-openshift-default-dockercfg-4hlzd\"" Nov 02 20:34:28.279100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.275948 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.280565 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.280498 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.281109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.280980 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.369994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.354320 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.371038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.371003 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lbk\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\") pod \"e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.471600 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.471448 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:28.890928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.890186 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 20:34:28.897644 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:28.897608 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aae6438_56d6_4650_ab1b_97e08f1cded8.slice/crio-fef50137a41abd08033b4c495ca0065df73106aa3b817283b408468680756d78 WatchSource:0}: Error finding container fef50137a41abd08033b4c495ca0065df73106aa3b817283b408468680756d78: Status 404 returned error can't find the container with id fef50137a41abd08033b4c495ca0065df73106aa3b817283b408468680756d78 Nov 02 20:34:28.901025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.900983 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:28.901123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.901104 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:28.901188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:28.901157 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:29.046152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.046117 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:29.046796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.046577 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:29.047591 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.047569 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:34:29.740313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.740277 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerStarted","Data":"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749"} Nov 02 20:34:29.740493 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.740334 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerStarted","Data":"fef50137a41abd08033b4c495ca0065df73106aa3b817283b408468680756d78"} Nov 02 20:34:29.949374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:29.949325 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podStartSLOduration=1.949310971 podStartE2EDuration="1.949310971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:29.900477787 +0000 UTC m=+2753.507022284" watchObservedRunningTime="2026-11-02 20:34:29.949310971 +0000 UTC m=+2753.555855469" Nov 02 20:34:30.471842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:30.471810 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:30.477497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:30.477475 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:30.746931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:30.746847 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:30.747898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:30.747873 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:34:31.633228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.633197 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 20:34:31.634350 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.634327 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:31.634416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.634354 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce6098e-9322-459c-85ae-ea2f58b8fd5f" containerName="istio-proxy" Nov 02 20:34:31.638115 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.638092 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.681376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.681337 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 20:34:31.687919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.687889 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-76c687\"" Nov 02 20:34:31.688576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.688551 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-76c687-dockercfg-7jzvv\"" Nov 02 20:34:31.715784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.715723 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.715784 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.715783 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnv4\" (UniqueName: \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.817185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.817154 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnv4\" (UniqueName: \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.817390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.817371 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.818066 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.818046 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.870813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.870781 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnv4\" (UniqueName: \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\") pod \"payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:31.952567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:31.952542 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:32.320844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:32.320817 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 20:34:32.321820 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:32.321786 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82af6c97_0661_4765_b9ee_b3928848fece.slice/crio-b682fcbc93fb3c9d99380d10ac7ff758b08a59906fe0c24bae926d49a81427ad WatchSource:0}: Error finding container b682fcbc93fb3c9d99380d10ac7ff758b08a59906fe0c24bae926d49a81427ad: Status 404 returned error can't find the container with id b682fcbc93fb3c9d99380d10ac7ff758b08a59906fe0c24bae926d49a81427ad Nov 02 20:34:32.762831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:32.762796 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" event={"ID":"82af6c97-0661-4765-b9ee-b3928848fece","Type":"ContainerStarted","Data":"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2"} Nov 02 20:34:32.763407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:32.762858 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" event={"ID":"82af6c97-0661-4765-b9ee-b3928848fece","Type":"ContainerStarted","Data":"b682fcbc93fb3c9d99380d10ac7ff758b08a59906fe0c24bae926d49a81427ad"} Nov 02 20:34:32.763407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:32.762915 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:32.878628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:32.878573 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" podStartSLOduration=1.878560377 podStartE2EDuration="1.878560377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:32.876502359 +0000 UTC m=+2756.483046859" watchObservedRunningTime="2026-11-02 20:34:32.878560377 +0000 UTC m=+2756.485104936" Nov 02 20:34:36.594399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.594365 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 20:34:36.598797 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.598775 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.682820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.682787 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.683001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.682952 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.683062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.683036 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97pc\" (UniqueName: \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.683186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.683169 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.695912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.695894 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2c6ce9-serving-cert\"" Nov 02 20:34:36.698670 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.698650 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 20:34:36.784265 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.784228 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.784460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.784271 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97pc\" (UniqueName: \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.784460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.784337 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.784460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.784448 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.784655 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:36.784471 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-serving-cert: secret "maas-api-e2e-shared-a-w2-2c6ce9-serving-cert" not found Nov 02 20:34:36.784655 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:36.784543 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls podName:6e94ace9-883e-4886-82e6-b8547bcdee50 nodeName:}" failed. No retries permitted until 2026-11-02 20:34:37.284520872 +0000 UTC m=+2760.891065408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls") pod "maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" (UID: "6e94ace9-883e-4886-82e6-b8547bcdee50") : secret "maas-api-e2e-shared-a-w2-2c6ce9-serving-cert" not found Nov 02 20:34:36.784974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.784951 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.786958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.786934 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:36.838924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:36.838889 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97pc\" (UniqueName: \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:37.291511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:37.291482 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:37.294217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:37.294191 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:37.513120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:37.513078 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:37.890017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:37.889985 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 20:34:37.891091 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:37.891057 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e94ace9_883e_4886_82e6_b8547bcdee50.slice/crio-caafcf166c6c59eb581e6fc0e4be51e0bf73ce02be9731690b02adb8abaf0fff WatchSource:0}: Error finding container caafcf166c6c59eb581e6fc0e4be51e0bf73ce02be9731690b02adb8abaf0fff: Status 404 returned error can't find the container with id caafcf166c6c59eb581e6fc0e4be51e0bf73ce02be9731690b02adb8abaf0fff Nov 02 20:34:38.802794 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:38.802637 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" event={"ID":"6e94ace9-883e-4886-82e6-b8547bcdee50","Type":"ContainerStarted","Data":"caafcf166c6c59eb581e6fc0e4be51e0bf73ce02be9731690b02adb8abaf0fff"} Nov 02 20:34:39.810132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:39.810097 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" event={"ID":"6e94ace9-883e-4886-82e6-b8547bcdee50","Type":"ContainerStarted","Data":"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a"} Nov 02 20:34:39.810539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:39.810167 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:43.771268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:43.771240 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 20:34:43.880177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:43.880110 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" podStartSLOduration=6.975108453 podStartE2EDuration="7.880096155s" totalImagesPullingTime="904.987702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:39.92105577 +0000 UTC m=+2763.527600269" watchObservedRunningTime="2026-11-02 20:34:43.880096155 +0000 UTC m=+2767.486640652" Nov 02 20:34:45.821773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:45.821720 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 20:34:46.978808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:46.978779 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:34:46.983300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:46.983280 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.059440 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:34:47.115013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.114982 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115044 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115145 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115181 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphjv\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115206 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115363 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115476 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115541 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.115612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.115590 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.216947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.216910 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.216947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.216949 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.216980 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217016 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217062 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217091 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphjv\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217123 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217197 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217287 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217393 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217491 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217531 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.217606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.217554 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.218056 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.218037 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.219558 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.219536 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.219743 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.219709 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.270434 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.270379 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.271121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.271098 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphjv\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.296936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.296917 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:47.711314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.711285 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:34:47.711623 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:34:47.711577 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef9e673_c881_4648_a321_1960812c7ed5.slice/crio-63c232d46fa7f178cd231cc7b27c961388dd276215315669860e62ec30ab95d9 WatchSource:0}: Error finding container 63c232d46fa7f178cd231cc7b27c961388dd276215315669860e62ec30ab95d9: Status 404 returned error can't find the container with id 63c232d46fa7f178cd231cc7b27c961388dd276215315669860e62ec30ab95d9 Nov 02 20:34:47.714345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.714308 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:47.714440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.714374 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:47.714440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.714405 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:34:47.864308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.864268 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" event={"ID":"8ef9e673-c881-4648-a321-1960812c7ed5","Type":"ContainerStarted","Data":"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee"} Nov 02 20:34:47.864514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.864334 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" event={"ID":"8ef9e673-c881-4648-a321-1960812c7ed5","Type":"ContainerStarted","Data":"63c232d46fa7f178cd231cc7b27c961388dd276215315669860e62ec30ab95d9"} Nov 02 20:34:47.977543 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:47.977431 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podStartSLOduration=1.977417322 podStartE2EDuration="1.977417322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:47.974104362 +0000 UTC m=+2771.580648862" watchObservedRunningTime="2026-11-02 20:34:47.977417322 +0000 UTC m=+2771.583961819" Nov 02 20:34:48.298037 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:48.297955 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:48.299245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:48.299218 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:48.299377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:48.299276 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:49.307755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:49.305146 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:49.307755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:49.305213 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:50.306163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:50.306124 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:50.306469 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:50.306439 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:51.025077 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:34:51.025039 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:34:51.300818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:51.300154 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:51.300818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:51.300217 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:52.309755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:52.302090 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:52.309755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:52.302165 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:53.307753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:53.307181 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:53.307753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:53.307247 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:54.297254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:54.297221 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 20:34:54.297628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:54.297272 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 20:34:55.302079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:55.302050 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:55.302502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:55.302466 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:34:55.303229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:34:55.303206 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:35:14.497953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:14.497915 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:35:14.498421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:14.498011 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:35:17.077629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.077598 5136 generic.go:356] "Generic (PLEG): container finished" podID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerID="f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843" exitCode=0 Nov 02 20:35:17.078088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.077674 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerDied","Data":"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843"} Nov 02 20:35:17.078134 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.078087 5136 scope.go:122] "RemoveContainer" containerID="f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843" Nov 02 20:35:17.079534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.079502 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:17.079621 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.079573 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:17.079621 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:17.079613 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:18.085850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:18.085817 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerStarted","Data":"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c"} Nov 02 20:35:18.864284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:18.864250 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:35:18.864517 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:18.864426 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:35:18.864517 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:18.864470 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 20:35:19.865198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:19.865167 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:35:19.865599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:19.865217 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 20:35:20.864455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:20.864418 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:35:20.864650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:20.864471 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 20:35:21.026531 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:35:21.026493 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:35:21.566611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.566573 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 20:35:21.571060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.571036 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.622864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.622834 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 20:35:21.625276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.625248 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.625404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.625297 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.625448 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.625418 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.625529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.625515 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrbl\" (UniqueName: \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.666918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.666891 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-76c687-serving-cert\"" Nov 02 20:35:21.726538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.726499 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.726758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.726560 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.726758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.726630 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.726758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.726706 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrbl\" (UniqueName: \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.727998 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:35:21.727621 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-serving-cert: secret "maas-api-e2e-shared-a-w0-76c687-serving-cert" not found Nov 02 20:35:21.727998 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:35:21.727704 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls podName:34e506f6-598b-480f-876e-e7cbb08ab632 nodeName:}" failed. No retries permitted until 2026-11-02 20:35:22.227680851 +0000 UTC m=+2805.834225331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls") pod "maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" (UID: "34e506f6-598b-480f-876e-e7cbb08ab632") : secret "maas-api-e2e-shared-a-w0-76c687-serving-cert" not found Nov 02 20:35:21.728768 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.728177 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.739228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.739189 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.780774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.780750 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrbl\" (UniqueName: \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:21.865523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.865440 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:35:21.865523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:21.865504 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 20:35:22.235065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.235028 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:22.237492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.237469 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:22.484668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.484636 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:22.864847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.864767 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:35:22.864847 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.864819 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 20:35:22.874221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.874196 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 20:35:22.874970 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:35:22.874944 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e506f6_598b_480f_876e_e7cbb08ab632.slice/crio-b7c0bfd43ef40b8e083a4d8b95d19f594edef625c47c5a781d8afb23db3d13f1 WatchSource:0}: Error finding container b7c0bfd43ef40b8e083a4d8b95d19f594edef625c47c5a781d8afb23db3d13f1: Status 404 returned error can't find the container with id b7c0bfd43ef40b8e083a4d8b95d19f594edef625c47c5a781d8afb23db3d13f1 Nov 02 20:35:22.876415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:22.876394 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:35:23.138751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:23.138641 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" event={"ID":"34e506f6-598b-480f-876e-e7cbb08ab632","Type":"ContainerStarted","Data":"b7c0bfd43ef40b8e083a4d8b95d19f594edef625c47c5a781d8afb23db3d13f1"} Nov 02 20:35:23.868850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:23.868820 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:35:23.869221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:23.869108 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:35:23.869786 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:23.869764 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 20:35:24.150969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:24.150936 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" event={"ID":"34e506f6-598b-480f-876e-e7cbb08ab632","Type":"ContainerStarted","Data":"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b"} Nov 02 20:35:24.151169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:24.151018 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:24.258804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:24.258724 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" podStartSLOduration=2.452768592 podStartE2EDuration="3.258706049s" totalImagesPullingTime="805.937457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:24.258495104 +0000 UTC m=+2807.865039602" watchObservedRunningTime="2026-11-02 20:35:24.258706049 +0000 UTC m=+2807.865250547" Nov 02 20:35:31.168388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:35:31.168359 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 20:35:51.004408 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:35:51.004364 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:36:13.567573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.567542 5136 generic.go:356] "Generic (PLEG): container finished" podID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerID="42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749" exitCode=0 Nov 02 20:36:13.568142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.567619 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerDied","Data":"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749"} Nov 02 20:36:13.568218 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.568192 5136 scope.go:122] "RemoveContainer" containerID="42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749" Nov 02 20:36:13.569693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.569664 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:13.569819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.569745 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:13.569819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:13.569811 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:14.575243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:14.575212 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerStarted","Data":"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6"} Nov 02 20:36:15.471799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:15.471769 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:36:15.472560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:15.472539 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 20:36:15.472619 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:15.472584 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 20:36:15.747306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:15.747233 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:36:16.472589 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:16.472556 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 20:36:16.472823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:16.472622 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 20:36:17.472298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:17.472262 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 20:36:17.472686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:17.472314 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 20:36:18.472066 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:18.472034 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 20:36:18.472259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:18.472085 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 20:36:19.471910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:19.471877 5136 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 20:36:19.472303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:19.471929 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 20:36:20.477462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:20.477431 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:36:20.478604 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:20.478583 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 20:36:21.003553 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:36:21.003524 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:36:30.953602 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:30.953558 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 20:36:30.957389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:30.957337 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.007039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.007009 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6b12ae-openshift-default-dockercfg-h2x9m\"" Nov 02 20:36:31.022343 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022361 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022483 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022536 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022663 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022642 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022711 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022877 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.022976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022957 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtph\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.023021 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.022997 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.057928 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.057902 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 20:36:31.123979 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.123947 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.123987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124012 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124057 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124149 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124185 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtph\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124203 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124248 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124283 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124389 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124494 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124641 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.124710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.124702 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.125071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.125047 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.126416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.126395 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.126778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.126756 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.179534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.179461 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtph\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.237096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.218077 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\") pod \"e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.271129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.271061 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:31.705019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.704991 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 20:36:31.705475 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:36:31.705450 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b06940_d7c2_4a9b_a302_9421de92d310.slice/crio-6760b56330a25dd6ef94fa400cf7e18104818b6bce8d7aafa6258097c7cd45bd WatchSource:0}: Error finding container 6760b56330a25dd6ef94fa400cf7e18104818b6bce8d7aafa6258097c7cd45bd: Status 404 returned error can't find the container with id 6760b56330a25dd6ef94fa400cf7e18104818b6bce8d7aafa6258097c7cd45bd Nov 02 20:36:31.708301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.708269 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:31.708388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.708332 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:31.708388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:31.708363 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:36:32.715502 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:32.715472 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerStarted","Data":"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5"} Nov 02 20:36:32.716017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:32.715518 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerStarted","Data":"6760b56330a25dd6ef94fa400cf7e18104818b6bce8d7aafa6258097c7cd45bd"} Nov 02 20:36:33.271635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:33.271601 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:33.277516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:33.277494 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:33.389200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:33.389147 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podStartSLOduration=3.389133542 podStartE2EDuration="3.389133542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:32.92016404 +0000 UTC m=+2876.526708535" watchObservedRunningTime="2026-11-02 20:36:33.389133542 +0000 UTC m=+2876.995678113" Nov 02 20:36:33.721805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:33.721775 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:33.722643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:33.722619 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:36:43.774646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:43.774614 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:36:47.008023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.007777 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 20:36:47.020993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.020958 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.049985 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.049953 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 20:36:47.093523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.093494 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6b12ae\"" Nov 02 20:36:47.093523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.093501 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6b12ae-dockercfg-wvc7v\"" Nov 02 20:36:47.114429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.114397 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.114572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.114525 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kntj5\" (UniqueName: \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.215677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.215640 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.215881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.215765 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kntj5\" (UniqueName: \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.216374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.216355 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.269867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.269799 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kntj5\" (UniqueName: \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\") pod \"payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.344256 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.344227 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:47.737267 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:36:47.737232 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc913b4_df32_4c31_bae1_b06377d1bb9b.slice/crio-c8b4301dd65c6da7982a52714fc4e97f726e0e7d649b4a7844947687b87202b6 WatchSource:0}: Error finding container c8b4301dd65c6da7982a52714fc4e97f726e0e7d649b4a7844947687b87202b6: Status 404 returned error can't find the container with id c8b4301dd65c6da7982a52714fc4e97f726e0e7d649b4a7844947687b87202b6 Nov 02 20:36:47.738026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.738002 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 20:36:47.834815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.834786 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" event={"ID":"cbc913b4-df32-4c31-bae1-b06377d1bb9b","Type":"ContainerStarted","Data":"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88"} Nov 02 20:36:47.834927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.834843 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" event={"ID":"cbc913b4-df32-4c31-bae1-b06377d1bb9b","Type":"ContainerStarted","Data":"c8b4301dd65c6da7982a52714fc4e97f726e0e7d649b4a7844947687b87202b6"} Nov 02 20:36:47.834927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:47.834907 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:51.004111 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:36:51.004061 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:36:51.356780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.356665 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" podStartSLOduration=5.356651068 podStartE2EDuration="5.356651068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:47.941990377 +0000 UTC m=+2891.548534875" watchObservedRunningTime="2026-11-02 20:36:51.356651068 +0000 UTC m=+2894.963195565" Nov 02 20:36:51.358439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.358419 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 20:36:51.362040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.362017 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.415543 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.415512 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 20:36:51.459007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.458975 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-578636\"" Nov 02 20:36:51.459223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.459166 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-578636-dockercfg-dj6lk\"" Nov 02 20:36:51.471936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.471826 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7qc\" (UniqueName: \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.472129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.471974 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.573689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.573651 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7qc\" (UniqueName: \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.573974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.573778 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.574488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.574466 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.628195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.628110 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7qc\" (UniqueName: \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\") pod \"payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:51.674925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:51.674898 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:52.043220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:52.043197 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 20:36:52.044064 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:36:52.043868 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aaf7ec_3461_4e97_9a5a_dec550bc45ec.slice/crio-24634f51669f8a85d1816aba334a9a440c03e8f8a39440960aafcf3d9429a6b8 WatchSource:0}: Error finding container 24634f51669f8a85d1816aba334a9a440c03e8f8a39440960aafcf3d9429a6b8: Status 404 returned error can't find the container with id 24634f51669f8a85d1816aba334a9a440c03e8f8a39440960aafcf3d9429a6b8 Nov 02 20:36:52.870242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:52.870205 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" event={"ID":"88aaf7ec-3461-4e97-9a5a-dec550bc45ec","Type":"ContainerStarted","Data":"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5"} Nov 02 20:36:52.870447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:52.870269 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:36:52.870447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:52.870285 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" event={"ID":"88aaf7ec-3461-4e97-9a5a-dec550bc45ec","Type":"ContainerStarted","Data":"24634f51669f8a85d1816aba334a9a440c03e8f8a39440960aafcf3d9429a6b8"} Nov 02 20:36:52.982281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:52.982233 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" podStartSLOduration=1.982219457 podStartE2EDuration="1.982219457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:52.979593404 +0000 UTC m=+2896.586137897" watchObservedRunningTime="2026-11-02 20:36:52.982219457 +0000 UTC m=+2896.588763954" Nov 02 20:36:58.799989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:58.799952 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" podUID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" containerName="main" containerID="cri-o://94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad" gracePeriod=45 Nov 02 20:36:58.846449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:58.846428 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 20:36:59.093913 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.093888 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:36:59.176922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.176894 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\" (UniqueName: \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.176936 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177027 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177093 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177237 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177261 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:59.177314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177271 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\") pod \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\" (UID: \"0536ef81-e3b4-4e74-8904-3aa71db8f9ad\") " Nov 02 20:36:59.177531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177508 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:59.177619 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.177530 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:59.178096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.178069 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.178212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.178100 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.178212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.178116 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.179501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.179468 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:59.179846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.179823 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:59.179939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.179841 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8" pod "0536ef81-e3b4-4e74-8904-3aa71db8f9ad" (UID: "0536ef81-e3b4-4e74-8904-3aa71db8f9ad"). InnerVolumeSpecName "kube-api-access-th9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:59.279023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.278992 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.279023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.279021 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.279023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.279031 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th9l8\" (UniqueName: \"kubernetes.io/projected/0536ef81-e3b4-4e74-8904-3aa71db8f9ad-kube-api-access-th9l8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.914588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.914557 5136 generic.go:356] "Generic (PLEG): container finished" podID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" containerID="94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad" exitCode=0 Nov 02 20:36:59.915006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.914600 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" Nov 02 20:36:59.915006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.914626 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" event={"ID":"0536ef81-e3b4-4e74-8904-3aa71db8f9ad","Type":"ContainerDied","Data":"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad"} Nov 02 20:36:59.915006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.914661 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm" event={"ID":"0536ef81-e3b4-4e74-8904-3aa71db8f9ad","Type":"ContainerDied","Data":"2e0b31c249e0e5f232a540f04a93fe0ab3b77b7716d95d87ff8c2fdf85bf6094"} Nov 02 20:36:59.915006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.914678 5136 scope.go:122] "RemoveContainer" containerID="94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad" Nov 02 20:36:59.925389 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.925369 5136 scope.go:122] "RemoveContainer" containerID="94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad" Nov 02 20:36:59.925682 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:36:59.925664 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad\": container with ID starting with 94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad not found: ID does not exist" containerID="94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad" Nov 02 20:36:59.925756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:36:59.925692 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad"} err="failed to get container status \"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad\": rpc error: code = NotFound desc = could not find container \"94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad\": container with ID starting with 94e2ba31a7722d281df085db21fe611046c1f7a38a8b317734953980193f0fad not found: ID does not exist" Nov 02 20:37:00.069882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:00.069854 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:37:00.073010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:00.072985 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-b743f6/praxis-route-b743f6-kserve-7478b6768f-58qqm"] Nov 02 20:37:00.547007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:00.546974 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" path="/var/lib/kubelet/pods/0536ef81-e3b4-4e74-8904-3aa71db8f9ad/volumes" Nov 02 20:37:03.879385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:03.879354 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 20:37:14.352998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.352962 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 20:37:14.353597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.353583 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" containerName="main" Nov 02 20:37:14.353638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.353598 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" containerName="main" Nov 02 20:37:14.353766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.353755 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0536ef81-e3b4-4e74-8904-3aa71db8f9ad" containerName="main" Nov 02 20:37:14.356716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.356693 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.410507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.410476 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 20:37:14.453915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.453870 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-09ecd0-serving-cert\"" Nov 02 20:37:14.473994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.473965 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxqs\" (UniqueName: \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.474135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.474039 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.474135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.474086 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.474210 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.474200 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.574999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.574970 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxqs\" (UniqueName: \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.575209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.575027 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.575209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.575193 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.575329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.575317 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.575329 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:14.575322 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-serving-cert: secret "maas-api-e2e-shared-b-w1-09ecd0-serving-cert" not found Nov 02 20:37:14.575401 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:14.575384 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls podName:1dab15d7-86a1-45a7-804c-348e82f90833 nodeName:}" failed. No retries permitted until 2026-11-02 20:37:15.075369858 +0000 UTC m=+2918.681914333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls") pod "maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" (UID: "1dab15d7-86a1-45a7-804c-348e82f90833") : secret "maas-api-e2e-shared-b-w1-09ecd0-serving-cert" not found Nov 02 20:37:14.576073 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.576052 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.577435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.577415 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:14.629988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:14.629931 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxqs\" (UniqueName: \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:15.086537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:15.086508 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:15.089045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:15.089019 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:15.269664 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:15.269630 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:15.638680 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:15.638652 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 20:37:15.639115 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:37:15.638950 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dab15d7_86a1_45a7_804c_348e82f90833.slice/crio-d8f06648876fd7bea236b7dcf6d1beb51b3e7dbaab12a3780b0dda1e4e3ef34e WatchSource:0}: Error finding container d8f06648876fd7bea236b7dcf6d1beb51b3e7dbaab12a3780b0dda1e4e3ef34e: Status 404 returned error can't find the container with id d8f06648876fd7bea236b7dcf6d1beb51b3e7dbaab12a3780b0dda1e4e3ef34e Nov 02 20:37:16.015280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:16.015238 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" event={"ID":"1dab15d7-86a1-45a7-804c-348e82f90833","Type":"ContainerStarted","Data":"d8f06648876fd7bea236b7dcf6d1beb51b3e7dbaab12a3780b0dda1e4e3ef34e"} Nov 02 20:37:17.023664 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:17.023632 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" event={"ID":"1dab15d7-86a1-45a7-804c-348e82f90833","Type":"ContainerStarted","Data":"b090f78e78012c33da11aaadcf8db5263269c1a17523e65ee254aeb661798dba"} Nov 02 20:37:17.024053 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:17.023763 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:17.135531 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:17.135477 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" podStartSLOduration=2.269150013 podStartE2EDuration="3.135465108s" totalImagesPullingTime="866.315095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:17.133960838 +0000 UTC m=+2920.740505347" watchObservedRunningTime="2026-11-02 20:37:17.135465108 +0000 UTC m=+2920.742009653" Nov 02 20:37:21.009072 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:21.009020 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:37:21.177384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:21.177353 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:37:21.277804 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:21.277715 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:37:23.035603 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:23.035577 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 20:37:26.178074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:26.178036 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" containerID="cri-o://0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad" gracePeriod=25 Nov 02 20:37:26.278308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:26.278267 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" containerID="cri-o://3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861" gracePeriod=25 Nov 02 20:37:32.791159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:32.791123 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:37:32.791676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:32.791178 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:37:42.791016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:42.790981 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:37:42.791016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:42.791001 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:37:44.112159 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.112119 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 20:37:44.117743 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.117715 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.169974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.169942 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 20:37:44.174131 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.174108 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.174277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.174244 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.174277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.174272 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg4s\" (UniqueName: \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.174380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.174313 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.212489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.212464 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6b12ae-serving-cert\"" Nov 02 20:37:44.275420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.275388 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.275625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.275489 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.275625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.275508 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg4s\" (UniqueName: \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.275625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.275549 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.275625 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:44.275558 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-serving-cert: secret "maas-api-e2e-shared-b-w0-6b12ae-serving-cert" not found Nov 02 20:37:44.275920 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:44.275633 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls podName:173712fc-991a-44ee-8a1a-23bed389a699 nodeName:}" failed. No retries permitted until 2026-11-02 20:37:44.775611639 +0000 UTC m=+2948.382156138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" (UID: "173712fc-991a-44ee-8a1a-23bed389a699") : secret "maas-api-e2e-shared-b-w0-6b12ae-serving-cert" not found Nov 02 20:37:44.276416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.276394 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.279017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.278987 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.330239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.330210 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg4s\" (UniqueName: \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.782319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.782290 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:44.784770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:44.784746 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:45.031496 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:45.031458 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:45.403639 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:37:45.403602 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173712fc_991a_44ee_8a1a_23bed389a699.slice/crio-c456b17e4d6aa3f73ca533b3a0f6c15805c569d037ea5f0521c551e97163ac4b WatchSource:0}: Error finding container c456b17e4d6aa3f73ca533b3a0f6c15805c569d037ea5f0521c551e97163ac4b: Status 404 returned error can't find the container with id c456b17e4d6aa3f73ca533b3a0f6c15805c569d037ea5f0521c551e97163ac4b Nov 02 20:37:45.414661 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:45.414635 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 20:37:46.220585 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.220553 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" event={"ID":"173712fc-991a-44ee-8a1a-23bed389a699","Type":"ContainerStarted","Data":"c456b17e4d6aa3f73ca533b3a0f6c15805c569d037ea5f0521c551e97163ac4b"} Nov 02 20:37:46.475914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.475881 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:37:46.505929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.505046 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\" (UniqueName: \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\") pod \"9ca63ae0-57c6-451c-8051-c3df1ea60256\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " Nov 02 20:37:46.505929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.505308 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\") pod \"9ca63ae0-57c6-451c-8051-c3df1ea60256\" (UID: \"9ca63ae0-57c6-451c-8051-c3df1ea60256\") " Nov 02 20:37:46.505929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.505848 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume" pod "9ca63ae0-57c6-451c-8051-c3df1ea60256" (UID: "9ca63ae0-57c6-451c-8051-c3df1ea60256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:46.507467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.507436 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr" pod "9ca63ae0-57c6-451c-8051-c3df1ea60256" (UID: "9ca63ae0-57c6-451c-8051-c3df1ea60256"). InnerVolumeSpecName "kube-api-access-h4ljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:46.567140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.567118 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:37:46.606993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.606962 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\") pod \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " Nov 02 20:37:46.607140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.607103 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\" (UniqueName: \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\") pod \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\" (UID: \"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b\") " Nov 02 20:37:46.607308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.607285 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume" pod "2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" (UID: "2ca00e7e-1d9f-435a-b017-53e4fef0cd8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:46.607809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.607787 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4ljr\" (UniqueName: \"kubernetes.io/projected/9ca63ae0-57c6-451c-8051-c3df1ea60256-kube-api-access-h4ljr\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.607809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.607806 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.607992 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.607815 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca63ae0-57c6-451c-8051-c3df1ea60256-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.609267 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.609240 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x" pod "2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" (UID: "2ca00e7e-1d9f-435a-b017-53e4fef0cd8b"). InnerVolumeSpecName "kube-api-access-dlt7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:46.709511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:46.709481 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlt7x\" (UniqueName: \"kubernetes.io/projected/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b-kube-api-access-dlt7x\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:47.275023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.274986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" event={"ID":"173712fc-991a-44ee-8a1a-23bed389a699","Type":"ContainerStarted","Data":"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1"} Nov 02 20:37:47.275223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.275060 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:47.276472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.276445 5136 generic.go:356] "Generic (PLEG): container finished" podID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerID="0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad" exitCode=0 Nov 02 20:37:47.276598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.276480 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" Nov 02 20:37:47.276598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.276529 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" event={"ID":"9ca63ae0-57c6-451c-8051-c3df1ea60256","Type":"ContainerDied","Data":"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad"} Nov 02 20:37:47.276598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.276575 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f" event={"ID":"9ca63ae0-57c6-451c-8051-c3df1ea60256","Type":"ContainerDied","Data":"83280878fbde227481d3d146ad8e48e7481f5240072637ecfb02e3052b297084"} Nov 02 20:37:47.276724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.276599 5136 scope.go:122] "RemoveContainer" containerID="0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad" Nov 02 20:37:47.277867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.277848 5136 generic.go:356] "Generic (PLEG): container finished" podID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerID="3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861" exitCode=0 Nov 02 20:37:47.277940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.277874 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" event={"ID":"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b","Type":"ContainerDied","Data":"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861"} Nov 02 20:37:47.277940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.277927 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" event={"ID":"2ca00e7e-1d9f-435a-b017-53e4fef0cd8b","Type":"ContainerDied","Data":"05a521a04d37037392a859f7a0ed64684b48feac1b237df45636a351200a124b"} Nov 02 20:37:47.278112 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.277886 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t" Nov 02 20:37:47.286987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.286972 5136 scope.go:122] "RemoveContainer" containerID="0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad" Nov 02 20:37:47.287259 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:47.287241 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad\": container with ID starting with 0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad not found: ID does not exist" containerID="0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad" Nov 02 20:37:47.287334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.287264 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad"} err="failed to get container status \"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad\": rpc error: code = NotFound desc = could not find container \"0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad\": container with ID starting with 0470a9c3e5597d566cf597a8d0ab6580b0fa1d88510b2afbbd7c335e056cbcad not found: ID does not exist" Nov 02 20:37:47.287334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.287278 5136 scope.go:122] "RemoveContainer" containerID="3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861" Nov 02 20:37:47.295916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.295901 5136 scope.go:122] "RemoveContainer" containerID="3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861" Nov 02 20:37:47.296174 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:47.296120 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861\": container with ID starting with 3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861 not found: ID does not exist" containerID="3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861" Nov 02 20:37:47.296268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.296178 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861"} err="failed to get container status \"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861\": rpc error: code = NotFound desc = could not find container \"3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861\": container with ID starting with 3f66476eccd41865a54ebabe59b89d067a7e3b2b9f5dd73c444fd3930e5b5861 not found: ID does not exist" Nov 02 20:37:47.388511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.388464 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" podStartSLOduration=2.57437897 podStartE2EDuration="3.388452115s" totalImagesPullingTime="814.073145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:47.385768018 +0000 UTC m=+2950.992312515" watchObservedRunningTime="2026-11-02 20:37:47.388452115 +0000 UTC m=+2950.994996613" Nov 02 20:37:47.492319 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.492291 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:37:47.495614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.495590 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b743f6-55b8444d67-vdd6f"] Nov 02 20:37:47.599613 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.599525 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:37:47.603549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:47.603520 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b743f6-7784c6d975-mdx4t"] Nov 02 20:37:48.546999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:48.546970 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" path="/var/lib/kubelet/pods/2ca00e7e-1d9f-435a-b017-53e4fef0cd8b/volumes" Nov 02 20:37:48.547391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:48.547348 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" path="/var/lib/kubelet/pods/9ca63ae0-57c6-451c-8051-c3df1ea60256/volumes" Nov 02 20:37:49.253895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.253860 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:37:49.254617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254597 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" Nov 02 20:37:49.254692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254620 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" Nov 02 20:37:49.254692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254645 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" Nov 02 20:37:49.254692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254653 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" Nov 02 20:37:49.254851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254792 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca00e7e-1d9f-435a-b017-53e4fef0cd8b" containerName="payload-pre-processing" Nov 02 20:37:49.254851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.254803 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca63ae0-57c6-451c-8051-c3df1ea60256" containerName="payload-processing" Nov 02 20:37:49.260498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.260480 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.310568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.310546 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-09ecd0-openshift-default-dockercfg-ccwlp\"" Nov 02 20:37:49.342289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342248 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342300 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342383 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342509 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnv6z\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342561 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342673 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342763 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342802 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.342839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.342820 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.363097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.363068 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:37:49.444848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.444802 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnv6z\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.444955 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.445083 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.445184 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.445237 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.445266 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.445360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.445307 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447335 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447404 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447508 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447538 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447543 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.447579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.447563 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.448500 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.448305 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.454078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.454048 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.455625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.455598 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.500332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.500036 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.500332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.500276 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnv6z\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.573981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.573907 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:49.989074 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:37:49.989048 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f47f1f_f6eb_4500_b510_bd3125a278ce.slice/crio-312debf8c6b9b98748b44dd8311640e825709ba28400d6d87c5982eabdee6355 WatchSource:0}: Error finding container 312debf8c6b9b98748b44dd8311640e825709ba28400d6d87c5982eabdee6355: Status 404 returned error can't find the container with id 312debf8c6b9b98748b44dd8311640e825709ba28400d6d87c5982eabdee6355 Nov 02 20:37:49.989220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.989171 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:37:49.991791 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.991758 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:37:49.991876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.991825 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:37:49.991876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:49.991853 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:37:50.300453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:50.300365 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" event={"ID":"22f47f1f-f6eb-4500-b510-bd3125a278ce","Type":"ContainerStarted","Data":"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e"} Nov 02 20:37:50.300453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:50.300436 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" event={"ID":"22f47f1f-f6eb-4500-b510-bd3125a278ce","Type":"ContainerStarted","Data":"312debf8c6b9b98748b44dd8311640e825709ba28400d6d87c5982eabdee6355"} Nov 02 20:37:50.417750 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:50.417691 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podStartSLOduration=1.417673937 podStartE2EDuration="1.417673937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:50.414661961 +0000 UTC m=+2954.021206461" watchObservedRunningTime="2026-11-02 20:37:50.417673937 +0000 UTC m=+2954.024218437" Nov 02 20:37:50.574441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:50.574361 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:51.022667 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:37:51.022636 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:37:51.576532 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:51.576497 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 20:37:51.576986 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:51.576543 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 20:37:52.574649 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:52.574615 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 20:37:52.574879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:52.574681 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 20:37:53.313647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:53.313609 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 20:37:53.574625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:53.574541 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 20:37:53.574625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:53.574602 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 20:37:54.575060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:54.575026 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 20:37:54.575536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:54.575096 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 20:37:55.574780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:55.574749 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.574962 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:55.574797 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 20:37:56.579383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:56.579348 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:56.579897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:56.579714 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:37:56.580438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:37:56.580422 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:38:03.627837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:03.627802 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:38:03.628801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:03.628050 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" podUID="ce7be8d0-359f-4502-8f37-e593a2212291" containerName="maas-api" containerID="cri-o://9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34" gracePeriod=30 Nov 02 20:38:03.975649 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:03.975619 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:38:04.068115 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.068084 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\") pod \"ce7be8d0-359f-4502-8f37-e593a2212291\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " Nov 02 20:38:04.068337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.068185 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") pod \"ce7be8d0-359f-4502-8f37-e593a2212291\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " Nov 02 20:38:04.068337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.068234 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\") pod \"ce7be8d0-359f-4502-8f37-e593a2212291\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " Nov 02 20:38:04.068466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.068377 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\" (UniqueName: \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\") pod \"ce7be8d0-359f-4502-8f37-e593a2212291\" (UID: \"ce7be8d0-359f-4502-8f37-e593a2212291\") " Nov 02 20:38:04.069426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.069395 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca" pod "ce7be8d0-359f-4502-8f37-e593a2212291" (UID: "ce7be8d0-359f-4502-8f37-e593a2212291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:04.071435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.071407 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls" pod "ce7be8d0-359f-4502-8f37-e593a2212291" (UID: "ce7be8d0-359f-4502-8f37-e593a2212291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:04.072827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.072792 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw" pod "ce7be8d0-359f-4502-8f37-e593a2212291" (UID: "ce7be8d0-359f-4502-8f37-e593a2212291"). InnerVolumeSpecName "kube-api-access-mh6xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:04.073150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.073105 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs" pod "ce7be8d0-359f-4502-8f37-e593a2212291" (UID: "ce7be8d0-359f-4502-8f37-e593a2212291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:04.169862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.169829 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:04.169862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.169857 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7be8d0-359f-4502-8f37-e593a2212291-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:04.169862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.169868 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7be8d0-359f-4502-8f37-e593a2212291-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:04.169862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.169876 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6xw\" (UniqueName: \"kubernetes.io/projected/ce7be8d0-359f-4502-8f37-e593a2212291-kube-api-access-mh6xw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:04.430478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.430395 5136 generic.go:356] "Generic (PLEG): container finished" podID="ce7be8d0-359f-4502-8f37-e593a2212291" containerID="9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34" exitCode=0 Nov 02 20:38:04.430478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.430450 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" Nov 02 20:38:04.430700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.430478 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" event={"ID":"ce7be8d0-359f-4502-8f37-e593a2212291","Type":"ContainerDied","Data":"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34"} Nov 02 20:38:04.430700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.430572 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7" event={"ID":"ce7be8d0-359f-4502-8f37-e593a2212291","Type":"ContainerDied","Data":"828842e851c53b4b3547ab1c40cdf10b43ef3548267f0a8769f962870b1b0f26"} Nov 02 20:38:04.430700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.430601 5136 scope.go:122] "RemoveContainer" containerID="9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34" Nov 02 20:38:04.442122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.442099 5136 scope.go:122] "RemoveContainer" containerID="9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34" Nov 02 20:38:04.442377 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:04.442357 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34\": container with ID starting with 9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34 not found: ID does not exist" containerID="9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34" Nov 02 20:38:04.442425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.442386 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34"} err="failed to get container status \"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34\": rpc error: code = NotFound desc = could not find container \"9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34\": container with ID starting with 9211eadc3819e403a602526f1bbc53375fa0481a88998953adbc744837d44a34 not found: ID does not exist" Nov 02 20:38:04.586225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.586188 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:38:04.654752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:04.649944 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b743f6-5b44988f68-mggv7"] Nov 02 20:38:06.548186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:06.548152 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7be8d0-359f-4502-8f37-e593a2212291" path="/var/lib/kubelet/pods/ce7be8d0-359f-4502-8f37-e593a2212291/volumes" Nov 02 20:38:09.468313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.468281 5136 generic.go:356] "Generic (PLEG): container finished" podID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerID="bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5" exitCode=0 Nov 02 20:38:09.468833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.468352 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerDied","Data":"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5"} Nov 02 20:38:09.468918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.468875 5136 scope.go:122] "RemoveContainer" containerID="bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5" Nov 02 20:38:09.470679 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.470648 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:09.470772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.470722 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:09.470816 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:09.470772 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:10.477022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:10.476989 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerStarted","Data":"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716"} Nov 02 20:38:10.722825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:10.722789 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:38:10.723077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:10.723031 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" containerID="cri-o://f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae" gracePeriod=30 Nov 02 20:38:11.271272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:11.271243 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:38:11.271480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:11.271455 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:38:11.271539 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:11.271520 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:38:11.822399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:11.822364 5136 patch_prober.go:29] interesting pod/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:38:11.822826 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:11.822430 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:38:12.271888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:12.271855 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:38:12.272099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:12.271915 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:38:13.272278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:13.272246 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:38:13.272655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:13.272300 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:38:14.271655 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:14.271614 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:38:14.271898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:14.271677 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:38:15.271576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:15.271546 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:38:15.272000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:15.271612 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:38:16.029523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.029496 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:38:16.121720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121690 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.121945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121769 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.121945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121811 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.121945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121835 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.121945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121903 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.122157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.121950 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.122157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122090 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:16.122157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122105 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.122308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122184 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.122308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122108 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:16.122308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122273 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:16.122308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122302 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\") pod \"37941455-7fe1-442e-aadd-e8a00eb3c975\" (UID: \"37941455-7fe1-442e-aadd-e8a00eb3c975\") " Nov 02 20:38:16.122511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122122 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:16.122810 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122784 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:16.122939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.122874 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37941455-7fe1-442e-aadd-e8a00eb3c975-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.123059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.123034 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.123059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.123059 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.123260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.123074 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.124338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.124312 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "kube-api-access-fvjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:16.125373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.125296 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:38:16.125605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.125570 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:16.125710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.125692 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token" pod "37941455-7fe1-442e-aadd-e8a00eb3c975" (UID: "37941455-7fe1-442e-aadd-e8a00eb3c975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:16.224514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.224478 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.224514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.224513 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.224781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.224529 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37941455-7fe1-442e-aadd-e8a00eb3c975-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.224781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.224545 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjxv\" (UniqueName: \"kubernetes.io/projected/37941455-7fe1-442e-aadd-e8a00eb3c975-kube-api-access-fvjxv\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.224781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.224561 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37941455-7fe1-442e-aadd-e8a00eb3c975-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:16.277642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.277563 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:38:16.278079 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.277965 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:38:16.278820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.278798 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 20:38:16.551818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.551696 5136 generic.go:356] "Generic (PLEG): container finished" podID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerID="f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae" exitCode=0 Nov 02 20:38:16.551818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.551759 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" Nov 02 20:38:16.551818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.551761 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerDied","Data":"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae"} Nov 02 20:38:16.551818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.551817 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h" event={"ID":"37941455-7fe1-442e-aadd-e8a00eb3c975","Type":"ContainerDied","Data":"0395054a68350c7c039fa4ca50bd07035a6dc7fdfe3e35e3e4958b4793220c43"} Nov 02 20:38:16.552169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.551846 5136 scope.go:122] "RemoveContainer" containerID="f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae" Nov 02 20:38:16.566879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.566857 5136 scope.go:122] "RemoveContainer" containerID="777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f" Nov 02 20:38:16.576766 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.576749 5136 scope.go:122] "RemoveContainer" containerID="f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae" Nov 02 20:38:16.577019 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:16.577000 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae\": container with ID starting with f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae not found: ID does not exist" containerID="f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae" Nov 02 20:38:16.577065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.577027 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae"} err="failed to get container status \"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae\": rpc error: code = NotFound desc = could not find container \"f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae\": container with ID starting with f9c6440a7cf6526e8c7cc314bc9ab7f9fb63fde6f7ae1b820b2bc8ae17fcbbae not found: ID does not exist" Nov 02 20:38:16.577065 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.577044 5136 scope.go:122] "RemoveContainer" containerID="777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f" Nov 02 20:38:16.577289 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:16.577268 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f\": container with ID starting with 777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f not found: ID does not exist" containerID="777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f" Nov 02 20:38:16.577351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.577301 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f"} err="failed to get container status \"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f\": rpc error: code = NotFound desc = could not find container \"777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f\": container with ID starting with 777412b38f90510c1c545c91f2bba6af9508b98275ac69131edfc5b8bae28c4f not found: ID does not exist" Nov 02 20:38:16.707903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.707871 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:38:16.711278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:16.711254 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b743f6-openshift-default-5b48676cdf-4kj2h"] Nov 02 20:38:17.812463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:17.812424 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:38:18.547492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:18.547451 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" path="/var/lib/kubelet/pods/37941455-7fe1-442e-aadd-e8a00eb3c975/volumes" Nov 02 20:38:20.666177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:20.666137 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:38:20.666693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:20.666380 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" podUID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" containerName="maas-api" containerID="cri-o://dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1" gracePeriod=30 Nov 02 20:38:20.982910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:20.982886 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:38:21.005199 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:21.005168 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:38:21.104978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.104948 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\") pod \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " Nov 02 20:38:21.105193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.104992 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") pod \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " Nov 02 20:38:21.105193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.105040 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\") pod \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " Nov 02 20:38:21.105193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.105129 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\" (UniqueName: \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\") pod \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\" (UID: \"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e\") " Nov 02 20:38:21.105815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.105783 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca" pod "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" (UID: "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:21.108306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.108279 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs" pod "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" (UID: "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:21.108542 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.108517 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw" pod "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" (UID: "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e"). InnerVolumeSpecName "kube-api-access-gm9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:21.109912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.109881 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls" pod "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" (UID: "5f6ee98c-ac86-40c6-b281-1ee621cf2c9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:21.207645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.207558 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.207645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.207594 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.207645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.207610 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.207645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.207627 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9zw\" (UniqueName: \"kubernetes.io/projected/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e-kube-api-access-gm9zw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.591012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.590921 5136 generic.go:356] "Generic (PLEG): container finished" podID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" containerID="dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1" exitCode=0 Nov 02 20:38:21.591012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.590964 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" Nov 02 20:38:21.591221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.591009 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" event={"ID":"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e","Type":"ContainerDied","Data":"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1"} Nov 02 20:38:21.591221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.591067 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc" event={"ID":"5f6ee98c-ac86-40c6-b281-1ee621cf2c9e","Type":"ContainerDied","Data":"667620fee363d9b59ff64933a2816fa1a839b133a61eb66a90b97dff404f653b"} Nov 02 20:38:21.591221 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.591095 5136 scope.go:122] "RemoveContainer" containerID="dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1" Nov 02 20:38:21.602006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.601977 5136 scope.go:122] "RemoveContainer" containerID="dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1" Nov 02 20:38:21.602323 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:21.602299 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1\": container with ID starting with dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1 not found: ID does not exist" containerID="dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1" Nov 02 20:38:21.602395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.602335 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1"} err="failed to get container status \"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1\": rpc error: code = NotFound desc = could not find container \"dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1\": container with ID starting with dab77bfd9e8ee4881347f3fda918a895f8a0132983fd055df679d98dc89e94e1 not found: ID does not exist" Nov 02 20:38:21.747567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.747536 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:38:21.752101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:21.752078 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-920071-69bf77cc74-49wrc"] Nov 02 20:38:22.548861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:22.548829 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" path="/var/lib/kubelet/pods/5f6ee98c-ac86-40c6-b281-1ee621cf2c9e/volumes" Nov 02 20:38:22.814022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:22.813933 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" podUID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" containerName="payload-pre-processing" containerID="cri-o://d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64" gracePeriod=25 Nov 02 20:38:23.096909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.096880 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:38:23.249777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.249747 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\") pod \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " Nov 02 20:38:23.249942 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.249822 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\" (UniqueName: \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\") pod \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\" (UID: \"06025fd0-f3ae-4e71-974d-fc6d41f36b4e\") " Nov 02 20:38:23.250215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.250183 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume" pod "06025fd0-f3ae-4e71-974d-fc6d41f36b4e" (UID: "06025fd0-f3ae-4e71-974d-fc6d41f36b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:23.250383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.250367 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:23.251848 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.251823 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm" pod "06025fd0-f3ae-4e71-974d-fc6d41f36b4e" (UID: "06025fd0-f3ae-4e71-974d-fc6d41f36b4e"). InnerVolumeSpecName "kube-api-access-5s5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:23.286783 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.286726 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:38:23.287409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287392 5136 cpu_manager.go:414] "containerMap: removing container" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287411 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287426 5136 cpu_manager.go:414] "containerMap: removing container" podUID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" containerName="maas-api" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287434 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" containerName="maas-api" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287479 5136 cpu_manager.go:414] "containerMap: removing container" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287487 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287504 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ce7be8d0-359f-4502-8f37-e593a2212291" containerName="maas-api" Nov 02 20:38:23.287516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287513 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7be8d0-359f-4502-8f37-e593a2212291" containerName="maas-api" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287527 5136 cpu_manager.go:414] "containerMap: removing container" podUID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" containerName="payload-pre-processing" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287535 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" containerName="payload-pre-processing" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287687 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287701 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7be8d0-359f-4502-8f37-e593a2212291" containerName="maas-api" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287713 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" containerName="payload-pre-processing" Nov 02 20:38:23.287909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.287748 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6ee98c-ac86-40c6-b281-1ee621cf2c9e" containerName="maas-api" Nov 02 20:38:23.290600 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.290581 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.343878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.340620 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2c6ce9\"/\"openshift-service-ca.crt\"" Nov 02 20:38:23.343878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.340630 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2c6ce9\"/\"kube-root-ca.crt\"" Nov 02 20:38:23.343878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.341529 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2c6ce9\"/\"default-dockercfg-nz8p9\"" Nov 02 20:38:23.343878 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.343330 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2c6ce9\"/\"auth-test-model-2c6ce9-kserve-self-signed-certs\"" Nov 02 20:38:23.345569 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.345543 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:38:23.351344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.351306 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5xm\" (UniqueName: \"kubernetes.io/projected/06025fd0-f3ae-4e71-974d-fc6d41f36b4e-kube-api-access-5s5xm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:23.453249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453206 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psnfh\" (UniqueName: \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.453417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453278 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.453480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453412 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.453480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453450 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.453568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453554 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.453613 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.453599 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555318 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psnfh\" (UniqueName: \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555367 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555415 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555444 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555521 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555572 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.555896 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555869 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.556026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555916 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.556026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.555965 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.557865 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.557844 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.558103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.558084 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.605772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.605704 5136 generic.go:356] "Generic (PLEG): container finished" podID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" containerID="d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64" exitCode=0 Nov 02 20:38:23.605772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.605751 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" event={"ID":"06025fd0-f3ae-4e71-974d-fc6d41f36b4e","Type":"ContainerDied","Data":"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64"} Nov 02 20:38:23.605909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.605787 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" Nov 02 20:38:23.605909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.605813 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh" event={"ID":"06025fd0-f3ae-4e71-974d-fc6d41f36b4e","Type":"ContainerDied","Data":"0d001e3c491bb4c366b40407be89debd6c4e292a3265dd5418b2482ad0a3d9c4"} Nov 02 20:38:23.605909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.605868 5136 scope.go:122] "RemoveContainer" containerID="d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64" Nov 02 20:38:23.609790 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.609725 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psnfh\" (UniqueName: \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\") pod \"auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:23.616605 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.616589 5136 scope.go:122] "RemoveContainer" containerID="d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64" Nov 02 20:38:23.616915 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:23.616898 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64\": container with ID starting with d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64 not found: ID does not exist" containerID="d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64" Nov 02 20:38:23.616975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.616924 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64"} err="failed to get container status \"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64\": rpc error: code = NotFound desc = could not find container \"d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64\": container with ID starting with d75cecbf3a3e9b7f2969d3ec10508fc47dce755587fd64fdb9dd305fec6efb64 not found: ID does not exist" Nov 02 20:38:23.764269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.764239 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:38:23.767254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.767212 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-920071-795d65f4c5-v9tsh"] Nov 02 20:38:23.902703 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:23.902615 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:24.271537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.271510 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:38:24.273070 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:38:24.273038 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1999d0a7_88d0_4686_89d2_591fc33a7e81.slice/crio-c87981b8b6e5c7e2d36af3e2169bd6a2bf8468a3c0af474035206a62a2510522 WatchSource:0}: Error finding container c87981b8b6e5c7e2d36af3e2169bd6a2bf8468a3c0af474035206a62a2510522: Status 404 returned error can't find the container with id c87981b8b6e5c7e2d36af3e2169bd6a2bf8468a3c0af474035206a62a2510522 Nov 02 20:38:24.547614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.547532 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06025fd0-f3ae-4e71-974d-fc6d41f36b4e" path="/var/lib/kubelet/pods/06025fd0-f3ae-4e71-974d-fc6d41f36b4e/volumes" Nov 02 20:38:24.615912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.615878 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" event={"ID":"1999d0a7-88d0-4686-89d2-591fc33a7e81","Type":"ContainerStarted","Data":"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90"} Nov 02 20:38:24.615912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.615918 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" event={"ID":"1999d0a7-88d0-4686-89d2-591fc33a7e81","Type":"ContainerStarted","Data":"c87981b8b6e5c7e2d36af3e2169bd6a2bf8468a3c0af474035206a62a2510522"} Nov 02 20:38:24.724334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.724286 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" podStartSLOduration=1.724271854 podStartE2EDuration="1.724271854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:24.723312548 +0000 UTC m=+2988.329857045" watchObservedRunningTime="2026-11-02 20:38:24.724271854 +0000 UTC m=+2988.330816351" Nov 02 20:38:24.903352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.903286 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:24.942151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.942122 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:38:24.942342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:24.942318 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" containerID="cri-o://c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c" gracePeriod=30 Nov 02 20:38:30.123104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.123078 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:38:30.146857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.146830 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.146989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.146874 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.146989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.146940 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.146989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.146982 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147050 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147092 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147156 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147211 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:30.147249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147225 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147248 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:30.147400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147348 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\") pod \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\" (UID: \"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca\") " Nov 02 20:38:30.147400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147365 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:30.147400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147384 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:30.147640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147529 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:30.148001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147967 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.148001 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.147993 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.148172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.148008 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.148172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.148023 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.148172 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.148036 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.149481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.149459 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "kube-api-access-jxhmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:30.149585 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.149553 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:30.149882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.149856 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:38:30.150004 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.149898 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token" pod "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" (UID: "25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:30.249507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.249436 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.249507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.249462 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.249507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.249471 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.249507 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.249483 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhmd\" (UniqueName: \"kubernetes.io/projected/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca-kube-api-access-jxhmd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:30.660085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.660050 5136 generic.go:356] "Generic (PLEG): container finished" podID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerID="c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c" exitCode=0 Nov 02 20:38:30.660235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.660101 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" Nov 02 20:38:30.660235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.660135 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerDied","Data":"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c"} Nov 02 20:38:30.660235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.660191 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr" event={"ID":"25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca","Type":"ContainerDied","Data":"915dfc505a5ae619bb8b15189b4c4b9b15e5e99572a4ab4a912a004cf998db47"} Nov 02 20:38:30.660235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.660213 5136 scope.go:122] "RemoveContainer" containerID="c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c" Nov 02 20:38:30.670534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.670507 5136 scope.go:122] "RemoveContainer" containerID="fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f" Nov 02 20:38:30.679246 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.679227 5136 scope.go:122] "RemoveContainer" containerID="c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c" Nov 02 20:38:30.679488 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:30.679470 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c\": container with ID starting with c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c not found: ID does not exist" containerID="c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c" Nov 02 20:38:30.679544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.679498 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c"} err="failed to get container status \"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c\": rpc error: code = NotFound desc = could not find container \"c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c\": container with ID starting with c5c977a99e71a05aa5c99f7856e8978aa8724a06d01d247d62eee9f37df3665c not found: ID does not exist" Nov 02 20:38:30.679544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.679514 5136 scope.go:122] "RemoveContainer" containerID="fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f" Nov 02 20:38:30.679766 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:30.679742 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f\": container with ID starting with fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f not found: ID does not exist" containerID="fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f" Nov 02 20:38:30.679825 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.679775 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f"} err="failed to get container status \"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f\": rpc error: code = NotFound desc = could not find container \"fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f\": container with ID starting with fbaf2911d698b793d33a2ef63a07dcde1dfc8ff0937c2321ad30d4325c07462f not found: ID does not exist" Nov 02 20:38:30.818310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.818167 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:38:30.864444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:30.864414 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-920071-openshift-default-55d855d99c-fjjvr"] Nov 02 20:38:32.546519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:32.546476 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" path="/var/lib/kubelet/pods/25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca/volumes" Nov 02 20:38:33.903710 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:33.903680 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:33.916228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:33.916204 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:34.714203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:34.714174 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:38:47.846922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.846893 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:38:47.847594 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847577 5136 cpu_manager.go:414] "containerMap: removing container" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847595 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847620 5136 cpu_manager.go:414] "containerMap: removing container" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847625 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847722 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847746 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ced36e-bf60-4eb0-bcf7-0e4d6c9622ca" containerName="istio-proxy" Nov 02 20:38:47.847805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.847755 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="37941455-7fe1-442e-aadd-e8a00eb3c975" containerName="istio-proxy" Nov 02 20:38:47.851828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.851807 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.874822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.874798 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.874935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.874835 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.874935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.874905 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.874971 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.874999 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bknj\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.875080 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875140 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.875127 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.875188 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.875236 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.875209 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.901467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.901443 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-d12bcf-openshift-default-dockercfg-4wxtt\"" Nov 02 20:38:47.952882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.952856 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:38:47.976075 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976049 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976251 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976090 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976251 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976194 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976249 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976276 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bknj\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976381 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976372 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976450 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976485 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976512 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976596 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976511 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976927 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.976963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.976943 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.977168 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.977141 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.979025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.978995 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:47.979197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:47.979173 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:48.030935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.030902 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:48.031320 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.031285 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bknj\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\") pod \"e2e-explicit-d12bcf-openshift-default-7698958fc-n77km\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:48.164086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.163959 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:48.597689 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.597655 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:38:48.634207 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:38:48.634176 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0562e22_0877_4267_b221_7d6ad89c7885.slice/crio-1e0d3960bca9ed9e0c36a2e72b1b532dbda72c774451ea6033417e545071cc08 WatchSource:0}: Error finding container 1e0d3960bca9ed9e0c36a2e72b1b532dbda72c774451ea6033417e545071cc08: Status 404 returned error can't find the container with id 1e0d3960bca9ed9e0c36a2e72b1b532dbda72c774451ea6033417e545071cc08 Nov 02 20:38:48.636719 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.636681 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:48.636826 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.636778 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:48.636826 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.636813 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:38:48.792628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.792532 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerStarted","Data":"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437"} Nov 02 20:38:48.792628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.792596 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerStarted","Data":"1e0d3960bca9ed9e0c36a2e72b1b532dbda72c774451ea6033417e545071cc08"} Nov 02 20:38:48.906394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:48.906293 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podStartSLOduration=1.906275558 podStartE2EDuration="1.906275558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:48.901473736 +0000 UTC m=+3012.508018235" watchObservedRunningTime="2026-11-02 20:38:48.906275558 +0000 UTC m=+3012.512820057" Nov 02 20:38:49.164962 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:49.164873 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:49.170752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:49.170711 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:49.799070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:49.799032 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:49.800070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:49.800051 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:38:51.004756 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:38:51.004709 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:38:55.252235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.252206 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:38:55.255945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.255926 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.309677 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.309645 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:38:55.385751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.385663 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.392854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.386067 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-d12bcf-dockercfg-zr6pg\"" Nov 02 20:38:55.392854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.386181 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-d12bcf\"" Nov 02 20:38:55.392854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.385749 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpw6n\" (UniqueName: \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.413042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.413011 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:38:55.417015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.416993 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.468570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.468540 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:38:55.487852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.487827 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.488025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.487883 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.488025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.487978 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpw6n\" (UniqueName: \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.488165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.488033 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpd7\" (UniqueName: \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.488631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.488608 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.541673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.541616 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpw6n\" (UniqueName: \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\") pod \"payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.570009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.569986 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:55.589365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.589341 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.589488 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.589424 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpd7\" (UniqueName: \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.590248 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.590226 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.646441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.646412 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpd7\" (UniqueName: \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\") pod \"payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.745038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.745006 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:55.936874 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:55.936848 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:38:55.937002 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:38:55.936971 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9258a8d1_8e03_4cd5_9300_0bdea11c0f12.slice/crio-7c7a4de2d528dbef91520059b26cd2bd4d2dfe3377d0d3ed56e7eb80d2670c0a WatchSource:0}: Error finding container 7c7a4de2d528dbef91520059b26cd2bd4d2dfe3377d0d3ed56e7eb80d2670c0a: Status 404 returned error can't find the container with id 7c7a4de2d528dbef91520059b26cd2bd4d2dfe3377d0d3ed56e7eb80d2670c0a Nov 02 20:38:56.110641 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.110612 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:38:56.111062 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:38:56.111027 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01722a76_15d9_467a_9ffd_3bf975c5aca0.slice/crio-29517be4eb2a02e26ee8e9db36656e6d8862ceea43494d01f60fd6274c00a29e WatchSource:0}: Error finding container 29517be4eb2a02e26ee8e9db36656e6d8862ceea43494d01f60fd6274c00a29e: Status 404 returned error can't find the container with id 29517be4eb2a02e26ee8e9db36656e6d8862ceea43494d01f60fd6274c00a29e Nov 02 20:38:56.854410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.854371 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" event={"ID":"9258a8d1-8e03-4cd5-9300-0bdea11c0f12","Type":"ContainerStarted","Data":"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984"} Nov 02 20:38:56.854892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.854441 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:38:56.854892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.854464 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" event={"ID":"9258a8d1-8e03-4cd5-9300-0bdea11c0f12","Type":"ContainerStarted","Data":"7c7a4de2d528dbef91520059b26cd2bd4d2dfe3377d0d3ed56e7eb80d2670c0a"} Nov 02 20:38:56.855927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.855902 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" event={"ID":"01722a76-15d9-467a-9ffd-3bf975c5aca0","Type":"ContainerStarted","Data":"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee"} Nov 02 20:38:56.856059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.855945 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" event={"ID":"01722a76-15d9-467a-9ffd-3bf975c5aca0","Type":"ContainerStarted","Data":"29517be4eb2a02e26ee8e9db36656e6d8862ceea43494d01f60fd6274c00a29e"} Nov 02 20:38:56.856059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.855979 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:38:56.964194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:56.964141 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" podStartSLOduration=1.964126982 podStartE2EDuration="1.964126982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:56.961171745 +0000 UTC m=+3020.567716250" watchObservedRunningTime="2026-11-02 20:38:56.964126982 +0000 UTC m=+3020.570671477" Nov 02 20:38:57.071265 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:38:57.071215 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" podStartSLOduration=2.071201872 podStartE2EDuration="2.071201872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:57.067586168 +0000 UTC m=+3020.674130666" watchObservedRunningTime="2026-11-02 20:38:57.071201872 +0000 UTC m=+3020.677746370" Nov 02 20:39:07.865003 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:07.864972 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:39:07.865506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:07.865273 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:39:21.004583 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:39:21.004550 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:39:29.826959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.826925 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:39:29.832702 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.832680 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.884349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.884317 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:39:29.927178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.927148 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"openshift-service-ca.crt\"" Nov 02 20:39:29.927347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.927304 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"auth-test-model-44a484-kserve-self-signed-certs\"" Nov 02 20:39:29.934332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.934310 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"kube-root-ca.crt\"" Nov 02 20:39:29.934332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.934330 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-44a484\"/\"default-dockercfg-wfkr9\"" Nov 02 20:39:29.940128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940101 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.940240 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940224 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.940285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940262 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.940368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940355 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfq8\" (UniqueName: \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.940430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940415 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:29.940487 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:29.940473 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041417 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041496 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041563 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041593 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041657 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfq8\" (UniqueName: \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.041890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041680 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.042003 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041914 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.042003 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.041958 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.042099 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.042079 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.043896 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.043872 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.044222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.044204 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.097305 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.097237 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfq8\" (UniqueName: \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\") pod \"auth-test-model-44a484-kserve-949f95889-66jvs\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.146525 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.146496 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:30.515595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:30.515568 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:39:30.516677 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:39:30.516650 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ba40a86_c504_4f76_81f8_f0d9048715ba.slice/crio-4519de80c46091a8b02faf591db6dd019043beee0cdeb83b26470a75124fdb7e WatchSource:0}: Error finding container 4519de80c46091a8b02faf591db6dd019043beee0cdeb83b26470a75124fdb7e: Status 404 returned error can't find the container with id 4519de80c46091a8b02faf591db6dd019043beee0cdeb83b26470a75124fdb7e Nov 02 20:39:31.067417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:31.067384 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" event={"ID":"9ba40a86-c504-4f76-81f8-f0d9048715ba","Type":"ContainerStarted","Data":"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b"} Nov 02 20:39:31.067842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:31.067427 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" event={"ID":"9ba40a86-c504-4f76-81f8-f0d9048715ba","Type":"ContainerStarted","Data":"4519de80c46091a8b02faf591db6dd019043beee0cdeb83b26470a75124fdb7e"} Nov 02 20:39:31.147127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:31.147094 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:31.179214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:31.179164 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" podStartSLOduration=2.179150268 podStartE2EDuration="2.179150268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:39:31.175443321 +0000 UTC m=+3054.781987822" watchObservedRunningTime="2026-11-02 20:39:31.179150268 +0000 UTC m=+3054.785694766" Nov 02 20:39:40.147114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:40.147072 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:40.160361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:40.160331 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:41.148168 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:39:41.148139 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:39:51.003986 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:39:51.003900 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:40:03.290134 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.290098 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:40:03.296410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.296388 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.346901 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.346865 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:40:03.384495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.384458 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pl8h\" (UniqueName: \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.384681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.384625 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.384681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.384669 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.384800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.384700 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.390310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.390278 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-d12bcf-serving-cert\"" Nov 02 20:40:03.486016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.485987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.486196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.486039 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.486196 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:40:03.486131 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-serving-cert: secret "maas-api-e2e-explicit-d12bcf-serving-cert" not found Nov 02 20:40:03.486196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.486150 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.486316 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:40:03.486200 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls podName:6a43dd51-c292-4eaf-a69c-490e39ac4670 nodeName:}" failed. No retries permitted until 2026-11-02 20:40:03.986183757 +0000 UTC m=+3087.592728233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls") pod "maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" (UID: "6a43dd51-c292-4eaf-a69c-490e39ac4670") : secret "maas-api-e2e-explicit-d12bcf-serving-cert" not found Nov 02 20:40:03.486316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.486297 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pl8h\" (UniqueName: \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.486821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.486793 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.488505 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.488481 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.541148 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.541054 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pl8h\" (UniqueName: \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.994048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.994010 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:03.996463 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:03.996443 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") pod \"maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:04.210265 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:04.210228 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:04.580121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:04.580085 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:40:04.580945 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:40:04.580916 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a43dd51_c292_4eaf_a69c_490e39ac4670.slice/crio-3668255d5f61e5ca7f95bc4eb14fe76241b8d99c39524da61ebdccdab07992ec WatchSource:0}: Error finding container 3668255d5f61e5ca7f95bc4eb14fe76241b8d99c39524da61ebdccdab07992ec: Status 404 returned error can't find the container with id 3668255d5f61e5ca7f95bc4eb14fe76241b8d99c39524da61ebdccdab07992ec Nov 02 20:40:05.307601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:05.307560 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" event={"ID":"6a43dd51-c292-4eaf-a69c-490e39ac4670","Type":"ContainerStarted","Data":"3668255d5f61e5ca7f95bc4eb14fe76241b8d99c39524da61ebdccdab07992ec"} Nov 02 20:40:06.315226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:06.315190 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" event={"ID":"6a43dd51-c292-4eaf-a69c-490e39ac4670","Type":"ContainerStarted","Data":"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e"} Nov 02 20:40:06.315615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:06.315256 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:06.424638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:06.424583 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" podStartSLOduration=2.484548558 podStartE2EDuration="3.424569447s" totalImagesPullingTime="940.020889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:40:06.424014339 +0000 UTC m=+3090.030558838" watchObservedRunningTime="2026-11-02 20:40:06.424569447 +0000 UTC m=+3090.031113945" Nov 02 20:40:08.203888 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:08.203860 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:40:08.204277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:08.204062 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" containerID="cri-o://c7c7f894a20dcb12fbc6e5a0b7254f56b8ac2a332da6fea977a800060a0e8151" gracePeriod=30 Nov 02 20:40:12.327534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:12.327506 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:40:13.361897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.361863 5136 generic.go:356] "Generic (PLEG): container finished" podID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerID="c7c7f894a20dcb12fbc6e5a0b7254f56b8ac2a332da6fea977a800060a0e8151" exitCode=0 Nov 02 20:40:13.362377 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.361934 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" event={"ID":"c2239908-f444-408c-801a-ccb0d1bc39ed","Type":"ContainerDied","Data":"c7c7f894a20dcb12fbc6e5a0b7254f56b8ac2a332da6fea977a800060a0e8151"} Nov 02 20:40:13.468338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.468284 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:40:13.517350 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517321 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517455 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517551 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517535 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517594 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517674 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517676 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:13.517840 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517707 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517840 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517784 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517844 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.517943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517865 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:13.517943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517903 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\") pod \"c2239908-f444-408c-801a-ccb0d1bc39ed\" (UID: \"c2239908-f444-408c-801a-ccb0d1bc39ed\") " Nov 02 20:40:13.518087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.517973 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:13.518087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518015 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:13.518197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:13.518506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518487 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.518593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518508 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.518593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518521 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.518593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518533 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2239908-f444-408c-801a-ccb0d1bc39ed-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.518593 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.518546 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.519856 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.519835 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:13.519956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.519907 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "kube-api-access-6tjcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:13.520136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.520119 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:40:13.520197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.520139 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token" pod "c2239908-f444-408c-801a-ccb0d1bc39ed" (UID: "c2239908-f444-408c-801a-ccb0d1bc39ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:13.619641 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.619614 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tjcd\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-kube-api-access-6tjcd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.619641 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.619639 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.619818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.619650 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:13.619818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:13.619659 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2239908-f444-408c-801a-ccb0d1bc39ed-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.369408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:14.369372 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" event={"ID":"c2239908-f444-408c-801a-ccb0d1bc39ed","Type":"ContainerDied","Data":"754e40789ab7afbc2c7419d5f3d0d65d5825e0667e25f31ca1351ab225c71eed"} Nov 02 20:40:14.369408 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:14.369393 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg" Nov 02 20:40:14.369995 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:14.369430 5136 scope.go:122] "RemoveContainer" containerID="c7c7f894a20dcb12fbc6e5a0b7254f56b8ac2a332da6fea977a800060a0e8151" Nov 02 20:40:14.573657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:14.573627 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:40:14.600833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:14.600807 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-bjrtg"] Nov 02 20:40:16.548706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:16.548663 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" path="/var/lib/kubelet/pods/c2239908-f444-408c-801a-ccb0d1bc39ed/volumes" Nov 02 20:40:21.004297 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:40:21.004261 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:40:34.524720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.524678 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:40:34.525176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.524915 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" containerID="cri-o://02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee" gracePeriod=30 Nov 02 20:40:34.549645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.549620 5136 generic.go:356] "Generic (PLEG): container finished" podID="c0562e22-0877-4267-b221-7d6ad89c7885" containerID="8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437" exitCode=0 Nov 02 20:40:34.549806 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.549693 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerDied","Data":"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437"} Nov 02 20:40:34.550073 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.550059 5136 scope.go:122] "RemoveContainer" containerID="8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437" Nov 02 20:40:34.551486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.551452 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:40:34.551611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.551538 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:40:34.551676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.551621 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:40:34.798991 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:34.798958 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:40:35.558274 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:35.558240 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerStarted","Data":"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19"} Nov 02 20:40:36.164196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:36.164165 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:40:36.165259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:36.165239 5136 patch_prober.go:29] interesting pod/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 20:40:36.165345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:36.165282 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 20:40:37.165086 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:37.165051 5136 patch_prober.go:29] interesting pod/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 20:40:37.165477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:37.165102 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 20:40:38.164466 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:38.164433 5136 patch_prober.go:29] interesting pod/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 20:40:38.164645 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:38.164500 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 20:40:39.164556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.164522 5136 patch_prober.go:29] interesting pod/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 20:40:39.164976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.164575 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 20:40:39.838279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.838254 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:40:39.919181 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919146 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919195 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919269 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919314 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919374 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919344 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919441 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919524 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919581 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919634 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\") pod \"8ef9e673-c881-4648-a321-1960812c7ed5\" (UID: \"8ef9e673-c881-4648-a321-1960812c7ed5\") " Nov 02 20:40:39.919896 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919710 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:39.920177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.919994 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:39.920177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920032 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:39.920390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920202 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:39.920454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920391 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:39.920454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920440 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:39.920567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920457 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:39.920567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920472 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:39.920567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.920557 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:39.921717 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.921691 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:39.922495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.922474 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:40:39.922635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.922620 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "kube-api-access-nphjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:39.922781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:39.922761 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token" pod "8ef9e673-c881-4648-a321-1960812c7ed5" (UID: "8ef9e673-c881-4648-a321-1960812c7ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:40.022508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.022468 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef9e673-c881-4648-a321-1960812c7ed5-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:40.022508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.022505 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:40.022716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.022519 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef9e673-c881-4648-a321-1960812c7ed5-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:40.022716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.022533 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef9e673-c881-4648-a321-1960812c7ed5-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:40.022716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.022548 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphjv\" (UniqueName: \"kubernetes.io/projected/8ef9e673-c881-4648-a321-1960812c7ed5-kube-api-access-nphjv\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:40.165294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.164835 5136 patch_prober.go:29] interesting pod/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 20:40:40.165294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.164893 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 20:40:40.616227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.616189 5136 generic.go:356] "Generic (PLEG): container finished" podID="8ef9e673-c881-4648-a321-1960812c7ed5" containerID="02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee" exitCode=0 Nov 02 20:40:40.616410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.616257 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" Nov 02 20:40:40.616410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.616270 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" event={"ID":"8ef9e673-c881-4648-a321-1960812c7ed5","Type":"ContainerDied","Data":"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee"} Nov 02 20:40:40.616410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.616322 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw" event={"ID":"8ef9e673-c881-4648-a321-1960812c7ed5","Type":"ContainerDied","Data":"63c232d46fa7f178cd231cc7b27c961388dd276215315669860e62ec30ab95d9"} Nov 02 20:40:40.616410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.616343 5136 scope.go:122] "RemoveContainer" containerID="02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee" Nov 02 20:40:40.626263 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.626245 5136 scope.go:122] "RemoveContainer" containerID="02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee" Nov 02 20:40:40.626525 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:40:40.626504 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee\": container with ID starting with 02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee not found: ID does not exist" containerID="02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee" Nov 02 20:40:40.626567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.626532 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee"} err="failed to get container status \"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee\": rpc error: code = NotFound desc = could not find container \"02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee\": container with ID starting with 02fd7c3f46469f60d459f3d6b6d75527804db938da83b67be16e5ce69a1b86ee not found: ID does not exist" Nov 02 20:40:40.773322 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.773288 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:40:40.777067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:40.777040 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-nhsnw"] Nov 02 20:40:41.169330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.169302 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:40:41.169712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.169698 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:40:41.170303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.170289 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:40:41.385628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.385593 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:40:41.386244 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386229 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" Nov 02 20:40:41.386297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386247 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" Nov 02 20:40:41.386297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386260 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" Nov 02 20:40:41.386297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386265 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" Nov 02 20:40:41.386403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386381 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" containerName="istio-proxy" Nov 02 20:40:41.386403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.386389 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2239908-f444-408c-801a-ccb0d1bc39ed" containerName="istio-proxy" Nov 02 20:40:41.389837 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.389816 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.440265 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.440184 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-78bef7\"/\"kube-root-ca.crt\"" Nov 02 20:40:41.440437 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.440369 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-78bef7\"/\"auth-test-model-78bef7-kserve-self-signed-certs\"" Nov 02 20:40:41.440501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.440483 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-78bef7\"/\"default-dockercfg-7tw2g\"" Nov 02 20:40:41.440583 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.440566 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-78bef7\"/\"openshift-service-ca.crt\"" Nov 02 20:40:41.445084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.445052 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:40:41.551445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.551363 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.551819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.551793 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.553226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.553203 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.553449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.553432 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.553949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.553925 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxksv\" (UniqueName: \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.554266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.554240 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655488 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655707 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655574 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655707 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655595 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655721 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxksv\" (UniqueName: \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655801 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.655886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.655871 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.656634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.656609 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.656859 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.656702 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.657010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.656943 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.658475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.658455 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.658840 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.658824 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:41.709773 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:41.709745 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxksv\" (UniqueName: \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\") pod \"auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:42.004639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.004560 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:42.371033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.371005 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:40:42.371428 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:40:42.371307 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9412d8d0_946c_4e7d_bf72_b2a32ca505d7.slice/crio-8545a68bb38d8a709eed7fe2ebcbfc150631efffa322f64c8af19b8ec5742d91 WatchSource:0}: Error finding container 8545a68bb38d8a709eed7fe2ebcbfc150631efffa322f64c8af19b8ec5742d91: Status 404 returned error can't find the container with id 8545a68bb38d8a709eed7fe2ebcbfc150631efffa322f64c8af19b8ec5742d91 Nov 02 20:40:42.547709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.547676 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef9e673-c881-4648-a321-1960812c7ed5" path="/var/lib/kubelet/pods/8ef9e673-c881-4648-a321-1960812c7ed5/volumes" Nov 02 20:40:42.630802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.630697 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" event={"ID":"9412d8d0-946c-4e7d-bf72-b2a32ca505d7","Type":"ContainerStarted","Data":"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b"} Nov 02 20:40:42.630802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.630767 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" event={"ID":"9412d8d0-946c-4e7d-bf72-b2a32ca505d7","Type":"ContainerStarted","Data":"8545a68bb38d8a709eed7fe2ebcbfc150631efffa322f64c8af19b8ec5742d91"} Nov 02 20:40:42.739890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:42.739835 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" podStartSLOduration=1.739819213 podStartE2EDuration="1.739819213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:40:42.738169063 +0000 UTC m=+3126.344713563" watchObservedRunningTime="2026-11-02 20:40:42.739819213 +0000 UTC m=+3126.346363714" Nov 02 20:40:43.005697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:43.005666 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:51.004167 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:40:51.004131 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:40:52.005199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:52.005166 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:52.017554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:52.017528 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:40:52.705712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:40:52.705684 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:41:21.005160 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:41:21.005080 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:41:51.005683 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:41:51.005648 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:42:18.259446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.259405 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:42:18.264006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.263982 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.316954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.316921 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:42:18.359616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.359583 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d12bcf\"/\"openshift-service-ca.crt\"" Nov 02 20:42:18.359808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.359792 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d12bcf\"/\"kube-root-ca.crt\"" Nov 02 20:42:18.361032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.361016 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d12bcf\"/\"default-dockercfg-spdgg\"" Nov 02 20:42:18.361090 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.361021 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d12bcf\"/\"explicit-model-d12bcf-kserve-self-signed-certs\"" Nov 02 20:42:18.373882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.373855 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.374013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.373893 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.374013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.373921 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.374097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.374040 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.374097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.374076 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9mb\" (UniqueName: \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.374232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.374215 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.475775 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.475720 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.475953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.475926 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.475998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.475957 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.475998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.475987 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.476076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.476034 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.476139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.476074 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9mb\" (UniqueName: \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.476139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.476127 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.476333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.476307 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.476443 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.476425 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.478271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.478243 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.478619 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.478603 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.533298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.533226 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9mb\" (UniqueName: \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\") pod \"explicit-model-d12bcf-kserve-55567dc7cf-9qwxv\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.576104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.576068 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:18.947192 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:18.947164 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:42:18.947896 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:42:18.947865 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a51f8c1_6383_46ca_82bb_33e5fac249a3.slice/crio-bed6ea86322f984b7b43e86e6ad6e791054dc83aaa688d64f1ddea14d9e4235e WatchSource:0}: Error finding container bed6ea86322f984b7b43e86e6ad6e791054dc83aaa688d64f1ddea14d9e4235e: Status 404 returned error can't find the container with id bed6ea86322f984b7b43e86e6ad6e791054dc83aaa688d64f1ddea14d9e4235e Nov 02 20:42:19.246940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:19.246896 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" event={"ID":"3a51f8c1-6383-46ca-82bb-33e5fac249a3","Type":"ContainerStarted","Data":"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f"} Nov 02 20:42:19.246940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:19.246940 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" event={"ID":"3a51f8c1-6383-46ca-82bb-33e5fac249a3","Type":"ContainerStarted","Data":"bed6ea86322f984b7b43e86e6ad6e791054dc83aaa688d64f1ddea14d9e4235e"} Nov 02 20:42:19.576644 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:19.576563 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:21.004238 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:42:21.004201 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:42:28.576973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:28.576939 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:28.589357 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:28.589333 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:28.698993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:28.698942 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" podStartSLOduration=10.698929141 podStartE2EDuration="10.698929141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:19.354691677 +0000 UTC m=+3222.961236175" watchObservedRunningTime="2026-11-02 20:42:28.698929141 +0000 UTC m=+3232.305473638" Nov 02 20:42:29.324019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:29.323984 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:42:51.004781 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:42:51.004690 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:42:52.970468 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:42:52.970438 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:43:07.990958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:07.990900 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" podUID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" containerName="main" containerID="cri-o://af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f" gracePeriod=45 Nov 02 20:43:08.269002 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.268978 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:43:08.325192 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325157 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325370 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325327 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325471 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:08.325640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325593 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:08.325640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325624 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325696 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325811 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\" (UniqueName: \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325864 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\") pod \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\" (UID: \"3a51f8c1-6383-46ca-82bb-33e5fac249a3\") " Nov 02 20:43:08.325953 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.325899 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:08.326394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.326372 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.326394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.326392 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.326394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.326402 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.328294 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.328273 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:08.328460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.328436 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:08.328537 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.328470 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb" pod "3a51f8c1-6383-46ca-82bb-33e5fac249a3" (UID: "3a51f8c1-6383-46ca-82bb-33e5fac249a3"). InnerVolumeSpecName "kube-api-access-2d9mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:08.428081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.428049 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9mb\" (UniqueName: \"kubernetes.io/projected/3a51f8c1-6383-46ca-82bb-33e5fac249a3-kube-api-access-2d9mb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.428081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.428079 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a51f8c1-6383-46ca-82bb-33e5fac249a3-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.428081 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.428088 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a51f8c1-6383-46ca-82bb-33e5fac249a3-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:08.580203 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.580171 5136 generic.go:356] "Generic (PLEG): container finished" podID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" containerID="af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f" exitCode=0 Nov 02 20:43:08.580344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.580209 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" event={"ID":"3a51f8c1-6383-46ca-82bb-33e5fac249a3","Type":"ContainerDied","Data":"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f"} Nov 02 20:43:08.580344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.580217 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" Nov 02 20:43:08.580344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.580235 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv" event={"ID":"3a51f8c1-6383-46ca-82bb-33e5fac249a3","Type":"ContainerDied","Data":"bed6ea86322f984b7b43e86e6ad6e791054dc83aaa688d64f1ddea14d9e4235e"} Nov 02 20:43:08.580344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.580258 5136 scope.go:122] "RemoveContainer" containerID="af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f" Nov 02 20:43:08.589708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.589687 5136 scope.go:122] "RemoveContainer" containerID="af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f" Nov 02 20:43:08.590035 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:08.590014 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f\": container with ID starting with af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f not found: ID does not exist" containerID="af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f" Nov 02 20:43:08.590104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.590042 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f"} err="failed to get container status \"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f\": rpc error: code = NotFound desc = could not find container \"af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f\": container with ID starting with af2a6050f45d77bad7a8aa31f593344562df5c5905fe8266cc702921801f1d0f not found: ID does not exist" Nov 02 20:43:08.737380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.737350 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:43:08.740129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:08.740106 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-d12bcf/explicit-model-d12bcf-kserve-55567dc7cf-9qwxv"] Nov 02 20:43:10.547119 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:10.547084 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" path="/var/lib/kubelet/pods/3a51f8c1-6383-46ca-82bb-33e5fac249a3/volumes" Nov 02 20:43:18.660005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:18.659975 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:43:18.762472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:18.762437 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:43:21.004571 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:21.004535 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:43:23.660376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.660324 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" podUID="01722a76-15d9-467a-9ffd-3bf975c5aca0" containerName="payload-processing" containerID="cri-o://8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee" gracePeriod=25 Nov 02 20:43:23.763415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.763344 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" podUID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" containerName="payload-pre-processing" containerID="cri-o://604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984" gracePeriod=25 Nov 02 20:43:23.934253 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.933994 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:43:23.974383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.974355 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\") pod \"01722a76-15d9-467a-9ffd-3bf975c5aca0\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " Nov 02 20:43:23.974529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.974394 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\" (UniqueName: \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\") pod \"01722a76-15d9-467a-9ffd-3bf975c5aca0\" (UID: \"01722a76-15d9-467a-9ffd-3bf975c5aca0\") " Nov 02 20:43:23.974805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.974778 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume" pod "01722a76-15d9-467a-9ffd-3bf975c5aca0" (UID: "01722a76-15d9-467a-9ffd-3bf975c5aca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:23.975213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.975188 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01722a76-15d9-467a-9ffd-3bf975c5aca0-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:23.978884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:23.978847 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7" pod "01722a76-15d9-467a-9ffd-3bf975c5aca0" (UID: "01722a76-15d9-467a-9ffd-3bf975c5aca0"). InnerVolumeSpecName "kube-api-access-vlpd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:24.030629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.030609 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:43:24.076016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.075986 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\") pod \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " Nov 02 20:43:24.076176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.076146 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\" (UniqueName: \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\") pod \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\" (UID: \"9258a8d1-8e03-4cd5-9300-0bdea11c0f12\") " Nov 02 20:43:24.076373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.076351 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume" pod "9258a8d1-8e03-4cd5-9300-0bdea11c0f12" (UID: "9258a8d1-8e03-4cd5-9300-0bdea11c0f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:24.076649 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.076632 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:24.076709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.076654 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpd7\" (UniqueName: \"kubernetes.io/projected/01722a76-15d9-467a-9ffd-3bf975c5aca0-kube-api-access-vlpd7\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:24.078174 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.078155 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n" pod "9258a8d1-8e03-4cd5-9300-0bdea11c0f12" (UID: "9258a8d1-8e03-4cd5-9300-0bdea11c0f12"). InnerVolumeSpecName "kube-api-access-hpw6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:24.178346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.178260 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpw6n\" (UniqueName: \"kubernetes.io/projected/9258a8d1-8e03-4cd5-9300-0bdea11c0f12-kube-api-access-hpw6n\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:24.682613 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.682585 5136 generic.go:356] "Generic (PLEG): container finished" podID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" containerID="604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984" exitCode=0 Nov 02 20:43:24.682987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.682645 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" Nov 02 20:43:24.682987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.682640 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" event={"ID":"9258a8d1-8e03-4cd5-9300-0bdea11c0f12","Type":"ContainerDied","Data":"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984"} Nov 02 20:43:24.682987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.682854 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54" event={"ID":"9258a8d1-8e03-4cd5-9300-0bdea11c0f12","Type":"ContainerDied","Data":"7c7a4de2d528dbef91520059b26cd2bd4d2dfe3377d0d3ed56e7eb80d2670c0a"} Nov 02 20:43:24.682987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.682878 5136 scope.go:122] "RemoveContainer" containerID="604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984" Nov 02 20:43:24.684256 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.684235 5136 generic.go:356] "Generic (PLEG): container finished" podID="01722a76-15d9-467a-9ffd-3bf975c5aca0" containerID="8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee" exitCode=0 Nov 02 20:43:24.684347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.684276 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" Nov 02 20:43:24.684347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.684314 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" event={"ID":"01722a76-15d9-467a-9ffd-3bf975c5aca0","Type":"ContainerDied","Data":"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee"} Nov 02 20:43:24.684447 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.684359 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2" event={"ID":"01722a76-15d9-467a-9ffd-3bf975c5aca0","Type":"ContainerDied","Data":"29517be4eb2a02e26ee8e9db36656e6d8862ceea43494d01f60fd6274c00a29e"} Nov 02 20:43:24.694567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.694545 5136 scope.go:122] "RemoveContainer" containerID="604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984" Nov 02 20:43:24.694955 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:24.694925 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984\": container with ID starting with 604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984 not found: ID does not exist" containerID="604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984" Nov 02 20:43:24.695034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.694965 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984"} err="failed to get container status \"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984\": rpc error: code = NotFound desc = could not find container \"604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984\": container with ID starting with 604307a68bf88fb01a08db051e297f8ff2a6d37502b28a9f9b1d3727999ad984 not found: ID does not exist" Nov 02 20:43:24.695034 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.694989 5136 scope.go:122] "RemoveContainer" containerID="8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee" Nov 02 20:43:24.706772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.706603 5136 scope.go:122] "RemoveContainer" containerID="8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee" Nov 02 20:43:24.706978 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:24.706950 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee\": container with ID starting with 8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee not found: ID does not exist" containerID="8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee" Nov 02 20:43:24.707059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.706988 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee"} err="failed to get container status \"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee\": rpc error: code = NotFound desc = could not find container \"8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee\": container with ID starting with 8392717f34464cc8f6e9b87bc7785bcef3f914b1e0e8356e21a648bb575f3cee not found: ID does not exist" Nov 02 20:43:24.839000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.838924 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:43:24.886242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.886213 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d12bcf-588bc6689-jg2m2"] Nov 02 20:43:24.961441 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.961399 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:43:24.964096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:24.964074 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d12bcf-57988d865f-84x54"] Nov 02 20:43:26.547200 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:26.547165 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01722a76-15d9-467a-9ffd-3bf975c5aca0" path="/var/lib/kubelet/pods/01722a76-15d9-467a-9ffd-3bf975c5aca0/volumes" Nov 02 20:43:26.547623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:26.547609 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" path="/var/lib/kubelet/pods/9258a8d1-8e03-4cd5-9300-0bdea11c0f12/volumes" Nov 02 20:43:44.140338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.140303 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:43:44.140898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.140583 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" podUID="6a43dd51-c292-4eaf-a69c-490e39ac4670" containerName="maas-api" containerID="cri-o://9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e" gracePeriod=30 Nov 02 20:43:44.423740 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.423703 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:43:44.555881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.555847 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\") pod \"6a43dd51-c292-4eaf-a69c-490e39ac4670\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " Nov 02 20:43:44.556149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.556129 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\" (UniqueName: \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\") pod \"6a43dd51-c292-4eaf-a69c-490e39ac4670\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " Nov 02 20:43:44.556300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.556284 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\") pod \"6a43dd51-c292-4eaf-a69c-490e39ac4670\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " Nov 02 20:43:44.556485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.556471 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") pod \"6a43dd51-c292-4eaf-a69c-490e39ac4670\" (UID: \"6a43dd51-c292-4eaf-a69c-490e39ac4670\") " Nov 02 20:43:44.567136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.567060 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca" pod "6a43dd51-c292-4eaf-a69c-490e39ac4670" (UID: "6a43dd51-c292-4eaf-a69c-490e39ac4670"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.574076 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.574033 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls" pod "6a43dd51-c292-4eaf-a69c-490e39ac4670" (UID: "6a43dd51-c292-4eaf-a69c-490e39ac4670"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:44.574234 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.574125 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h" pod "6a43dd51-c292-4eaf-a69c-490e39ac4670" (UID: "6a43dd51-c292-4eaf-a69c-490e39ac4670"). InnerVolumeSpecName "kube-api-access-6pl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.575501 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.575474 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs" pod "6a43dd51-c292-4eaf-a69c-490e39ac4670" (UID: "6a43dd51-c292-4eaf-a69c-490e39ac4670"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:44.659180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.659104 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a43dd51-c292-4eaf-a69c-490e39ac4670-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.659820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.659797 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pl8h\" (UniqueName: \"kubernetes.io/projected/6a43dd51-c292-4eaf-a69c-490e39ac4670-kube-api-access-6pl8h\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.660064 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.659992 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.660232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.660069 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a43dd51-c292-4eaf-a69c-490e39ac4670-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.818413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.818375 5136 generic.go:356] "Generic (PLEG): container finished" podID="6a43dd51-c292-4eaf-a69c-490e39ac4670" containerID="9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e" exitCode=0 Nov 02 20:43:44.818563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.818420 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" Nov 02 20:43:44.818563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.818418 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" event={"ID":"6a43dd51-c292-4eaf-a69c-490e39ac4670","Type":"ContainerDied","Data":"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e"} Nov 02 20:43:44.818563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.818549 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd" event={"ID":"6a43dd51-c292-4eaf-a69c-490e39ac4670","Type":"ContainerDied","Data":"3668255d5f61e5ca7f95bc4eb14fe76241b8d99c39524da61ebdccdab07992ec"} Nov 02 20:43:44.818751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.818579 5136 scope.go:122] "RemoveContainer" containerID="9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e" Nov 02 20:43:44.831832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.831813 5136 scope.go:122] "RemoveContainer" containerID="9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e" Nov 02 20:43:44.832129 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:44.832113 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e\": container with ID starting with 9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e not found: ID does not exist" containerID="9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e" Nov 02 20:43:44.832177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.832135 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e"} err="failed to get container status \"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e\": rpc error: code = NotFound desc = could not find container \"9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e\": container with ID starting with 9dd58815e7e1d1618cbe401bac0828ed2ad700e1f98e0fda8fd96bea861a1c2e not found: ID does not exist" Nov 02 20:43:44.974636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.974608 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:43:44.980640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:44.980620 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d12bcf-59bd464cff-87vrd"] Nov 02 20:43:45.663483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:45.663452 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:43:45.663910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:45.663642 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" containerID="cri-o://79ed3cbda72d4af8c94ebd087ca8ae1d2253644b90e09af75e6bfa1cb5abdc1b" gracePeriod=30 Nov 02 20:43:46.547311 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:46.547278 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a43dd51-c292-4eaf-a69c-490e39ac4670" path="/var/lib/kubelet/pods/6a43dd51-c292-4eaf-a69c-490e39ac4670/volumes" Nov 02 20:43:48.268983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:48.268951 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:43:48.269386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:48.269173 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" containerID="cri-o://16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19" gracePeriod=30 Nov 02 20:43:50.539809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:50.539776 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:50.540201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:50.539826 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:50.859966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:50.859765 5136 generic.go:356] "Generic (PLEG): container finished" podID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerID="79ed3cbda72d4af8c94ebd087ca8ae1d2253644b90e09af75e6bfa1cb5abdc1b" exitCode=0 Nov 02 20:43:50.859966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:50.859876 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" event={"ID":"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4","Type":"ContainerDied","Data":"79ed3cbda72d4af8c94ebd087ca8ae1d2253644b90e09af75e6bfa1cb5abdc1b"} Nov 02 20:43:50.920993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:50.920967 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:43:51.003882 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:51.003849 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:43:51.052726 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.052699 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.052882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.052852 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.052936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.052888 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.052936 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.052923 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:51.053024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.052980 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053009 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053049 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053131 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053135 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:51.053306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053196 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053306 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053251 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\") pod \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\" (UID: \"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4\") " Nov 02 20:43:51.053432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053353 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:51.053648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053540 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:51.053751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053701 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:51.053879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053860 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.053956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053885 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.053956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053895 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.053956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053904 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.053956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.053913 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.055189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.055160 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:51.055277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.055238 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:51.055625 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.055606 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:51.055690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.055626 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs" pod "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" (UID: "3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4"). InnerVolumeSpecName "kube-api-access-gk8qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:51.154767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.154677 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.154767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.154705 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.154767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.154715 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk8qs\" (UniqueName: \"kubernetes.io/projected/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-kube-api-access-gk8qs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.154767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.154726 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:51.866679 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.866649 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" Nov 02 20:43:51.867184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.866688 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh" event={"ID":"3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4","Type":"ContainerDied","Data":"61fed7c8c0101f8b4385103ae39b758ec9eba3c3e151a2e4f16b5e6fb4bd010e"} Nov 02 20:43:51.867184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:51.866774 5136 scope.go:122] "RemoveContainer" containerID="79ed3cbda72d4af8c94ebd087ca8ae1d2253644b90e09af75e6bfa1cb5abdc1b" Nov 02 20:43:52.074387 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:52.074358 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:43:52.078495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:52.078465 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-7n2zh"] Nov 02 20:43:52.547013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:52.546982 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" path="/var/lib/kubelet/pods/3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4/volumes" Nov 02 20:43:53.534197 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.534145 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:43:53.686250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686222 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686269 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686326 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686371 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686405 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686489 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:53.686640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686575 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:53.686640 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686580 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686663 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.686814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686686 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:53.686814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686699 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:53.686814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686705 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.687012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686828 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\") pod \"c0562e22-0877-4267-b221-7d6ad89c7885\" (UID: \"c0562e22-0877-4267-b221-7d6ad89c7885\") " Nov 02 20:43:53.687012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.686858 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:53.687388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.687360 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.687522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.687390 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.687522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.687404 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0562e22-0877-4267-b221-7d6ad89c7885-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.687522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.687417 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.687522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.687430 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.688652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.688628 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:53.688898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.688880 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:53.688966 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.688933 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "kube-api-access-7bknj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:53.689006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.688970 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy" pod "c0562e22-0877-4267-b221-7d6ad89c7885" (UID: "c0562e22-0877-4267-b221-7d6ad89c7885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:53.788376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.788300 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0562e22-0877-4267-b221-7d6ad89c7885-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.788376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.788326 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0562e22-0877-4267-b221-7d6ad89c7885-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.788376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.788336 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bknj\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-kube-api-access-7bknj\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.788376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.788345 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0562e22-0877-4267-b221-7d6ad89c7885-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:53.881229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.881199 5136 generic.go:356] "Generic (PLEG): container finished" podID="c0562e22-0877-4267-b221-7d6ad89c7885" containerID="16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19" exitCode=0 Nov 02 20:43:53.881410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.881246 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" Nov 02 20:43:53.881410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.881276 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerDied","Data":"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19"} Nov 02 20:43:53.881410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.881333 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km" event={"ID":"c0562e22-0877-4267-b221-7d6ad89c7885","Type":"ContainerDied","Data":"1e0d3960bca9ed9e0c36a2e72b1b532dbda72c774451ea6033417e545071cc08"} Nov 02 20:43:53.881410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.881364 5136 scope.go:122] "RemoveContainer" containerID="16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19" Nov 02 20:43:53.893155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.893120 5136 scope.go:122] "RemoveContainer" containerID="8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437" Nov 02 20:43:53.901780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.901762 5136 scope.go:122] "RemoveContainer" containerID="16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19" Nov 02 20:43:53.902028 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:53.902012 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19\": container with ID starting with 16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19 not found: ID does not exist" containerID="16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19" Nov 02 20:43:53.902077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.902036 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19"} err="failed to get container status \"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19\": rpc error: code = NotFound desc = could not find container \"16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19\": container with ID starting with 16f940e68ebae3ff50ec0c094175a8b93c31d80ede5a8d8d71c5c07916723e19 not found: ID does not exist" Nov 02 20:43:53.902077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.902052 5136 scope.go:122] "RemoveContainer" containerID="8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437" Nov 02 20:43:53.902234 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:43:53.902219 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437\": container with ID starting with 8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437 not found: ID does not exist" containerID="8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437" Nov 02 20:43:53.902273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:53.902239 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437"} err="failed to get container status \"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437\": rpc error: code = NotFound desc = could not find container \"8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437\": container with ID starting with 8388ed9c99f0cb81b205d6214fb9f6fc04c3db6499195cffcdd12cad0455d437 not found: ID does not exist" Nov 02 20:43:54.038122 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:54.038092 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:43:54.040701 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:54.040623 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-d12bcf-openshift-default-7698958fc-n77km"] Nov 02 20:43:54.546899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:43:54.546862 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" path="/var/lib/kubelet/pods/c0562e22-0877-4267-b221-7d6ad89c7885/volumes" Nov 02 20:44:07.117020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.116989 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:44:07.117566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117552 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.117612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117567 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.117612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117577 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.117612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117582 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.117612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117603 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" Nov 02 20:44:07.117612 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117608 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117617 5136 cpu_manager.go:414] "containerMap: removing container" podUID="6a43dd51-c292-4eaf-a69c-490e39ac4670" containerName="maas-api" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117622 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a43dd51-c292-4eaf-a69c-490e39ac4670" containerName="maas-api" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117638 5136 cpu_manager.go:414] "containerMap: removing container" podUID="01722a76-15d9-467a-9ffd-3bf975c5aca0" containerName="payload-processing" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117642 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01722a76-15d9-467a-9ffd-3bf975c5aca0" containerName="payload-processing" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117651 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" containerName="payload-pre-processing" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117656 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" containerName="payload-pre-processing" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117667 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" containerName="main" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117671 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" containerName="main" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117780 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.117788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117789 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0562e22-0877-4267-b221-7d6ad89c7885" containerName="istio-proxy" Nov 02 20:44:07.118082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117802 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6fc5ad-1d5c-40ba-a339-3f0f2cc497c4" containerName="istio-proxy" Nov 02 20:44:07.118082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117811 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a43dd51-c292-4eaf-a69c-490e39ac4670" containerName="maas-api" Nov 02 20:44:07.118082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117822 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="01722a76-15d9-467a-9ffd-3bf975c5aca0" containerName="payload-processing" Nov 02 20:44:07.118082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117829 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9258a8d1-8e03-4cd5-9300-0bdea11c0f12" containerName="payload-pre-processing" Nov 02 20:44:07.118082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.117835 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a51f8c1-6383-46ca-82bb-33e5fac249a3" containerName="main" Nov 02 20:44:07.122899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.122877 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.174723 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.174681 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:44:07.217622 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.217593 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-09e014\"/\"kube-root-ca.crt\"" Nov 02 20:44:07.217811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.217635 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-09e014\"/\"openshift-service-ca.crt\"" Nov 02 20:44:07.217811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.217691 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-09e014\"/\"noten-model-09e014-kserve-self-signed-certs\"" Nov 02 20:44:07.220308 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.220283 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-09e014\"/\"default-dockercfg-rlplt\"" Nov 02 20:44:07.248147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248114 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcdg\" (UniqueName: \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.248281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248240 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.248341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248279 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.248383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248348 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.248383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248372 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.248470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.248402 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.351656 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351621 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcdg\" (UniqueName: \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.351858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351724 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.351858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351763 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.351858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351818 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.351858 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351849 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.352094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.351896 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.353007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.352978 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.353268 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.353241 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.353522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.353499 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.367819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.366483 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.367819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.366689 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.407461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.407425 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcdg\" (UniqueName: \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\") pod \"noten-model-09e014-kserve-6c4cf8869-m5g9h\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.435279 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.435249 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:07.813073 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.813051 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:44:07.815038 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:44:07.815005 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80fecd06_705f_438d_aec5_4a0c7b3f285a.slice/crio-78164f807677378b9e771b72aab15617ac76af0dca459261d815fbd5789e71f0 WatchSource:0}: Error finding container 78164f807677378b9e771b72aab15617ac76af0dca459261d815fbd5789e71f0: Status 404 returned error can't find the container with id 78164f807677378b9e771b72aab15617ac76af0dca459261d815fbd5789e71f0 Nov 02 20:44:07.968903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.968862 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" event={"ID":"80fecd06-705f-438d-aec5-4a0c7b3f285a","Type":"ContainerStarted","Data":"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3"} Nov 02 20:44:07.969109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:07.968917 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" event={"ID":"80fecd06-705f-438d-aec5-4a0c7b3f285a","Type":"ContainerStarted","Data":"78164f807677378b9e771b72aab15617ac76af0dca459261d815fbd5789e71f0"} Nov 02 20:44:08.079231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:08.077942 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" podStartSLOduration=1.07792423 podStartE2EDuration="1.07792423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:08.076794813 +0000 UTC m=+3331.683339311" watchObservedRunningTime="2026-11-02 20:44:08.07792423 +0000 UTC m=+3331.684468733" Nov 02 20:44:08.436464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:08.436381 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:17.435839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:17.435755 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:17.448709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:17.448683 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:18.041421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:18.041391 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:44:21.004686 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:44:21.004650 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:44:31.009217 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:31.009183 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:44:31.009636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:31.009417 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" containerID="cri-o://b159b455b923f6732a041fcabc78d1f1ee4d1af4b8815431985170520331e23f" gracePeriod=30 Nov 02 20:44:35.885117 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:35.885083 5136 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:44:35.885503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:35.885151 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:44:36.142017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.141936 5136 generic.go:356] "Generic (PLEG): container finished" podID="453911ab-1871-4618-832a-48cfdd10aa20" containerID="b159b455b923f6732a041fcabc78d1f1ee4d1af4b8815431985170520331e23f" exitCode=0 Nov 02 20:44:36.142017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.141977 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" event={"ID":"453911ab-1871-4618-832a-48cfdd10aa20","Type":"ContainerDied","Data":"b159b455b923f6732a041fcabc78d1f1ee4d1af4b8815431985170520331e23f"} Nov 02 20:44:36.285815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.285789 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:44:36.403633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403500 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403630 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403659 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403700 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403830 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403745 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403844 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.403957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403849 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:36.404040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403929 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:36.404040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.403982 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.404040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404032 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.404338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404066 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\") pod \"453911ab-1871-4618-832a-48cfdd10aa20\" (UID: \"453911ab-1871-4618-832a-48cfdd10aa20\") " Nov 02 20:44:36.404338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404203 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:36.404338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404264 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:36.404581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404434 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:36.404725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404706 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453911ab-1871-4618-832a-48cfdd10aa20-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.404811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404754 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.404811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404770 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.404811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404780 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.404811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.404789 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.406222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.406199 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "kube-api-access-t9xfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:36.406329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.406229 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:36.406329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.406250 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:36.406329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.406250 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo" pod "453911ab-1871-4618-832a-48cfdd10aa20" (UID: "453911ab-1871-4618-832a-48cfdd10aa20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:36.505716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.505686 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453911ab-1871-4618-832a-48cfdd10aa20-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.505716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.505713 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453911ab-1871-4618-832a-48cfdd10aa20-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.505716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.505723 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xfc\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-kube-api-access-t9xfc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:36.505971 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:36.505756 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453911ab-1871-4618-832a-48cfdd10aa20-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:37.149453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:37.149420 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" Nov 02 20:44:37.149972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:37.149458 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s" event={"ID":"453911ab-1871-4618-832a-48cfdd10aa20","Type":"ContainerDied","Data":"d4ac08fa9b6c89df521a1a9b3418467c923b67759cd87d0931234b56cf2eb55e"} Nov 02 20:44:37.149972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:37.149523 5136 scope.go:122] "RemoveContainer" containerID="b159b455b923f6732a041fcabc78d1f1ee4d1af4b8815431985170520331e23f" Nov 02 20:44:37.353566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:37.353539 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:44:37.401016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:37.400945 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-hng7s"] Nov 02 20:44:38.547475 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:38.547443 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453911ab-1871-4618-832a-48cfdd10aa20" path="/var/lib/kubelet/pods/453911ab-1871-4618-832a-48cfdd10aa20/volumes" Nov 02 20:44:49.893248 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:44:49.893215 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:44:51.003905 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:44:51.003873 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:45:04.917572 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:04.917531 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerName="main" containerID="cri-o://9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3" gracePeriod=45 Nov 02 20:45:05.030351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.030309 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": dial tcp 10.133.0.159:8000: connect: connection refused" Nov 02 20:45:05.198997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.198972 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:45:05.257630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257598 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\" (UniqueName: \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.257813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257687 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.257813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257760 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.257813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257788 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.257923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257867 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\" (UniqueName: \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.257993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257976 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\") pod \"80fecd06-705f-438d-aec5-4a0c7b3f285a\" (UID: \"80fecd06-705f-438d-aec5-4a0c7b3f285a\") " Nov 02 20:45:05.258058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.257970 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:05.258058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.258038 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:05.258176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.258079 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:05.258449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.258429 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.258449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.258447 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.258650 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.258458 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.260124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.260091 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:05.260225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.260136 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:05.260225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.260146 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg" pod "80fecd06-705f-438d-aec5-4a0c7b3f285a" (UID: "80fecd06-705f-438d-aec5-4a0c7b3f285a"). InnerVolumeSpecName "kube-api-access-qmcdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:05.334749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.329498 5136 generic.go:356] "Generic (PLEG): container finished" podID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerID="9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3" exitCode=0 Nov 02 20:45:05.334749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.329665 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" event={"ID":"80fecd06-705f-438d-aec5-4a0c7b3f285a","Type":"ContainerDied","Data":"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3"} Nov 02 20:45:05.334749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.329711 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" event={"ID":"80fecd06-705f-438d-aec5-4a0c7b3f285a","Type":"ContainerDied","Data":"78164f807677378b9e771b72aab15617ac76af0dca459261d815fbd5789e71f0"} Nov 02 20:45:05.334749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.329747 5136 scope.go:122] "RemoveContainer" containerID="9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3" Nov 02 20:45:05.334749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.329904 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h" Nov 02 20:45:05.350864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.350841 5136 scope.go:122] "RemoveContainer" containerID="9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3" Nov 02 20:45:05.351164 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:05.351141 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3\": container with ID starting with 9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3 not found: ID does not exist" containerID="9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3" Nov 02 20:45:05.351295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.351177 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3"} err="failed to get container status \"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3\": rpc error: code = NotFound desc = could not find container \"9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3\": container with ID starting with 9a86008340622631fd194ecbd5efe321eb218d9c3f3d9d8f65e97c7577abf3f3 not found: ID does not exist" Nov 02 20:45:05.360120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.360084 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fecd06-705f-438d-aec5-4a0c7b3f285a-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.360232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.360127 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fecd06-705f-438d-aec5-4a0c7b3f285a-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.360232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.360143 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcdg\" (UniqueName: \"kubernetes.io/projected/80fecd06-705f-438d-aec5-4a0c7b3f285a-kube-api-access-qmcdg\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:05.496442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.496411 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:45:05.504827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:05.504795 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-09e014/noten-model-09e014-kserve-6c4cf8869-m5g9h"] Nov 02 20:45:06.548648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:06.548611 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" path="/var/lib/kubelet/pods/80fecd06-705f-438d-aec5-4a0c7b3f285a/volumes" Nov 02 20:45:10.856870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:10.856839 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:45:15.578257 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.578228 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:45:15.815959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.815928 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-59ffc9"] Nov 02 20:45:15.816561 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816543 5136 cpu_manager.go:414] "containerMap: removing container" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" Nov 02 20:45:15.816636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816562 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" Nov 02 20:45:15.816636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816616 5136 cpu_manager.go:414] "containerMap: removing container" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerName="main" Nov 02 20:45:15.816636 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816624 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerName="main" Nov 02 20:45:15.816857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816814 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fecd06-705f-438d-aec5-4a0c7b3f285a" containerName="main" Nov 02 20:45:15.816892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.816871 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="453911ab-1871-4618-832a-48cfdd10aa20" containerName="istio-proxy" Nov 02 20:45:15.820709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.820692 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:15.869188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.869123 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-59ffc9"] Nov 02 20:45:15.872057 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.872030 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\"" Nov 02 20:45:15.898654 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.898622 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbv5\" (UniqueName: \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\") pod \"test-curl-7516-59ffc9\" (UID: \"be085875-f862-475d-809e-e674334755b4\") " pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:15.999379 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:15.999344 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbv5\" (UniqueName: \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\") pod \"test-curl-7516-59ffc9\" (UID: \"be085875-f862-475d-809e-e674334755b4\") " pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:16.053771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:16.053721 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbv5\" (UniqueName: \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\") pod \"test-curl-7516-59ffc9\" (UID: \"be085875-f862-475d-809e-e674334755b4\") " pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:16.132155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:16.132078 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:16.503819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:16.503789 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-59ffc9"] Nov 02 20:45:16.504952 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:16.504923 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe085875_f862_475d_809e_e674334755b4.slice/crio-1b44c9918a7dbb74258a4796a72f1a66a67649b701b37867f0f77ccc9b56518a WatchSource:0}: Error finding container 1b44c9918a7dbb74258a4796a72f1a66a67649b701b37867f0f77ccc9b56518a: Status 404 returned error can't find the container with id 1b44c9918a7dbb74258a4796a72f1a66a67649b701b37867f0f77ccc9b56518a Nov 02 20:45:17.405423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:17.405384 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-59ffc9" event={"ID":"be085875-f862-475d-809e-e674334755b4","Type":"ContainerStarted","Data":"5c638b2b97e99aecfe8b218667762292c8b488bf9cf9b36ed9cfd4c9bdf9d3cd"} Nov 02 20:45:17.405871 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:17.405432 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-59ffc9" event={"ID":"be085875-f862-475d-809e-e674334755b4","Type":"ContainerStarted","Data":"1b44c9918a7dbb74258a4796a72f1a66a67649b701b37867f0f77ccc9b56518a"} Nov 02 20:45:17.513439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:17.513383 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7516-59ffc9" podStartSLOduration=2.513371372 podStartE2EDuration="2.513371372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:17.510833577 +0000 UTC m=+3401.117378091" watchObservedRunningTime="2026-11-02 20:45:17.513371372 +0000 UTC m=+3401.119915866" Nov 02 20:45:19.029163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.029116 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-59ffc9"] Nov 02 20:45:19.031940 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.031913 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-59ffc9"] Nov 02 20:45:19.415352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.415255 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-9fcacc"] Nov 02 20:45:19.415924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.415903 5136 cpu_manager.go:414] "containerMap: removing container" podUID="be085875-f862-475d-809e-e674334755b4" containerName="test-curl-7516-59ffc9" Nov 02 20:45:19.415924 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.415921 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be085875-f862-475d-809e-e674334755b4" containerName="test-curl-7516-59ffc9" Nov 02 20:45:19.416121 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.416050 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="be085875-f862-475d-809e-e674334755b4" containerName="test-curl-7516-59ffc9" Nov 02 20:45:19.418777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.418751 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7516-59ffc9" podUID="be085875-f862-475d-809e-e674334755b4" containerName="test-curl-7516-59ffc9" containerID="cri-o://5c638b2b97e99aecfe8b218667762292c8b488bf9cf9b36ed9cfd4c9bdf9d3cd" gracePeriod=2 Nov 02 20:45:19.419692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.419673 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:19.468171 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:19.467243 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-59ffc9\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="be085875-f862-475d-809e-e674334755b4" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:19.468997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.468967 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-9fcacc"] Nov 02 20:45:19.515858 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:19.515823 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-59ffc9\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="be085875-f862-475d-809e-e674334755b4" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:19.539780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.539748 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9qf\" (UniqueName: \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\") pod \"test-curl-7516-9fcacc\" (UID: \"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43\") " pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:19.641628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.641595 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9qf\" (UniqueName: \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\") pod \"test-curl-7516-9fcacc\" (UID: \"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43\") " pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:19.697189 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.697159 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9qf\" (UniqueName: \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\") pod \"test-curl-7516-9fcacc\" (UID: \"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43\") " pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:19.731147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:19.731121 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:20.100149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:20.100116 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-9fcacc"] Nov 02 20:45:20.101057 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:20.101027 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73fd09c_b6a6_492f_8a9c_ba3b285c8e43.slice/crio-9f5e988515f9bc11bdf8beca979554352284a271195274505592cc424ab745b1 WatchSource:0}: Error finding container 9f5e988515f9bc11bdf8beca979554352284a271195274505592cc424ab745b1: Status 404 returned error can't find the container with id 9f5e988515f9bc11bdf8beca979554352284a271195274505592cc424ab745b1 Nov 02 20:45:20.427062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:20.426983 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-9fcacc" event={"ID":"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43","Type":"ContainerStarted","Data":"5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a"} Nov 02 20:45:20.427062 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:20.427024 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-9fcacc" event={"ID":"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43","Type":"ContainerStarted","Data":"9f5e988515f9bc11bdf8beca979554352284a271195274505592cc424ab745b1"} Nov 02 20:45:20.535167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:20.535112 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7516-9fcacc" podStartSLOduration=1.535097046 podStartE2EDuration="1.535097046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:20.534274873 +0000 UTC m=+3404.140819371" watchObservedRunningTime="2026-11-02 20:45:20.535097046 +0000 UTC m=+3404.141641562" Nov 02 20:45:20.582645 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:20.582609 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-59ffc9\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="be085875-f862-475d-809e-e674334755b4" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:21.004171 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:21.004139 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:45:21.434228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.434204 5136 generic.go:356] "Generic (PLEG): container finished" podID="be085875-f862-475d-809e-e674334755b4" containerID="5c638b2b97e99aecfe8b218667762292c8b488bf9cf9b36ed9cfd4c9bdf9d3cd" exitCode=137 Nov 02 20:45:21.624694 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.624671 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:21.769826 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.769800 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\" (UniqueName: \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\") pod \"be085875-f862-475d-809e-e674334755b4\" (UID: \"be085875-f862-475d-809e-e674334755b4\") " Nov 02 20:45:21.771914 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.771885 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5" pod "be085875-f862-475d-809e-e674334755b4" (UID: "be085875-f862-475d-809e-e674334755b4"). InnerVolumeSpecName "kube-api-access-jpbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:21.864067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.864041 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-9fcacc"] Nov 02 20:45:21.868974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.868944 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-9fcacc"] Nov 02 20:45:21.871504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.871481 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbv5\" (UniqueName: \"kubernetes.io/projected/be085875-f862-475d-809e-e674334755b4-kube-api-access-jpbv5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.992883 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.992853 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-c9c6b8"] Nov 02 20:45:21.993491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.993470 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" containerName="test-curl-7516-9fcacc" Nov 02 20:45:21.993491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.993487 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" containerName="test-curl-7516-9fcacc" Nov 02 20:45:21.993607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.993579 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" containerName="test-curl-7516-9fcacc" Nov 02 20:45:21.997032 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:21.997013 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:22.048144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.048088 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-c9c6b8"] Nov 02 20:45:22.175334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.175298 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7js\" (UniqueName: \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\") pod \"test-curl-7516-c9c6b8\" (UID: \"f0417b32-cae6-4006-8267-fff4faaaa1e6\") " pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:22.276093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.276057 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7js\" (UniqueName: \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\") pod \"test-curl-7516-c9c6b8\" (UID: \"f0417b32-cae6-4006-8267-fff4faaaa1e6\") " pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:22.330071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.329992 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7js\" (UniqueName: \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\") pod \"test-curl-7516-c9c6b8\" (UID: \"f0417b32-cae6-4006-8267-fff4faaaa1e6\") " pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:22.442014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.441982 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:22.442014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.442010 5136 scope.go:122] "RemoveContainer" containerID="5c638b2b97e99aecfe8b218667762292c8b488bf9cf9b36ed9cfd4c9bdf9d3cd" Nov 02 20:45:22.442552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.442280 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7516-9fcacc" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" containerName="test-curl-7516-9fcacc" containerID="cri-o://5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a" gracePeriod=2 Nov 02 20:45:22.490546 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:22.490513 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-9fcacc\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:22.538935 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:22.538903 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-59ffc9\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="be085875-f862-475d-809e-e674334755b4" pod="opendatahub/test-curl-7516-59ffc9" Nov 02 20:45:22.547409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.547384 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be085875-f862-475d-809e-e674334755b4" path="/var/lib/kubelet/pods/be085875-f862-475d-809e-e674334755b4/volumes" Nov 02 20:45:22.587223 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:22.587159 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-9fcacc\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:22.609025 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.609003 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:22.973660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:22.973632 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-c9c6b8"] Nov 02 20:45:22.974757 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:22.974716 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0417b32_cae6_4006_8267_fff4faaaa1e6.slice/crio-0a88a59ded545723193e9b7f1eb155a8c6c149e1986361ecfc32d158c3a79110 WatchSource:0}: Error finding container 0a88a59ded545723193e9b7f1eb155a8c6c149e1986361ecfc32d158c3a79110: Status 404 returned error can't find the container with id 0a88a59ded545723193e9b7f1eb155a8c6c149e1986361ecfc32d158c3a79110 Nov 02 20:45:23.449378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:23.449346 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-c9c6b8" event={"ID":"f0417b32-cae6-4006-8267-fff4faaaa1e6","Type":"ContainerStarted","Data":"be385127316f06c96adad53256db1ae044e339270064a4151f14d139149a7d5f"} Nov 02 20:45:23.449860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:23.449387 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-c9c6b8" event={"ID":"f0417b32-cae6-4006-8267-fff4faaaa1e6","Type":"ContainerStarted","Data":"0a88a59ded545723193e9b7f1eb155a8c6c149e1986361ecfc32d158c3a79110"} Nov 02 20:45:23.557489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:23.557442 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7516-c9c6b8" podStartSLOduration=2.55742916 podStartE2EDuration="2.55742916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:23.554840689 +0000 UTC m=+3407.161385184" watchObservedRunningTime="2026-11-02 20:45:23.55742916 +0000 UTC m=+3407.163973657" Nov 02 20:45:24.655085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:24.655064 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:24.805596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:24.805569 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\" (UniqueName: \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\") pod \"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43\" (UID: \"c73fd09c-b6a6-492f-8a9c-ba3b285c8e43\") " Nov 02 20:45:24.807545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:24.807523 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf" pod "c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" (UID: "c73fd09c-b6a6-492f-8a9c-ba3b285c8e43"). InnerVolumeSpecName "kube-api-access-6q9qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:24.907315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:24.907284 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9qf\" (UniqueName: \"kubernetes.io/projected/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43-kube-api-access-6q9qf\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:25.004486 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.004444 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-c9c6b8"] Nov 02 20:45:25.009211 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.008307 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-c9c6b8"] Nov 02 20:45:25.137557 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.137481 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-8d1941"] Nov 02 20:45:25.138134 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.138118 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" containerName="test-curl-7516-c9c6b8" Nov 02 20:45:25.138204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.138135 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" containerName="test-curl-7516-c9c6b8" Nov 02 20:45:25.138272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.138260 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" containerName="test-curl-7516-c9c6b8" Nov 02 20:45:25.141692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.141673 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:25.191251 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.191223 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-8d1941"] Nov 02 20:45:25.312887 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.312852 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fsb\" (UniqueName: \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\") pod \"test-curl-7516-8d1941\" (UID: \"18412f5a-861b-44fa-9d4b-7878aadc5400\") " pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:25.414372 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.414287 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fsb\" (UniqueName: \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\") pod \"test-curl-7516-8d1941\" (UID: \"18412f5a-861b-44fa-9d4b-7878aadc5400\") " pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:25.468285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.468252 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fsb\" (UniqueName: \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\") pod \"test-curl-7516-8d1941\" (UID: \"18412f5a-861b-44fa-9d4b-7878aadc5400\") " pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:25.471576 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.471544 5136 generic.go:356] "Generic (PLEG): container finished" podID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" containerID="5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a" exitCode=137 Nov 02 20:45:25.471687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.471600 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:25.471687 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.471629 5136 scope.go:122] "RemoveContainer" containerID="5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a" Nov 02 20:45:25.471988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.471957 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7516-c9c6b8" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" containerName="test-curl-7516-c9c6b8" containerID="cri-o://be385127316f06c96adad53256db1ae044e339270064a4151f14d139149a7d5f" gracePeriod=2 Nov 02 20:45:25.487989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.486941 5136 scope.go:122] "RemoveContainer" containerID="5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a" Nov 02 20:45:25.487989 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:25.487836 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a\": container with ID starting with 5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a not found: ID does not exist" containerID="5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a" Nov 02 20:45:25.487989 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.487871 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a"} err="failed to get container status \"5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a\": rpc error: code = NotFound desc = could not find container \"5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a\": container with ID starting with 5cb528c5f312ea9a0abee762a14d392c9f2d764e530e947932d634ac1382664a not found: ID does not exist" Nov 02 20:45:25.520445 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:25.520418 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:25.586755 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:25.585959 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-9fcacc\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:25.634317 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:25.634281 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:25.754530 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.754497 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:25.877238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:25.877178 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" podUID="1999d0a7-88d0-4686-89d2-591fc33a7e81" containerName="main" containerID="cri-o://2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90" gracePeriod=45 Nov 02 20:45:26.121127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.121101 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-8d1941"] Nov 02 20:45:26.122461 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:26.122377 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18412f5a_861b_44fa_9d4b_7878aadc5400.slice/crio-e249cbf311605aed3669d7315a5278bd9ab3501d45d391bb55398d1620b03edc WatchSource:0}: Error finding container e249cbf311605aed3669d7315a5278bd9ab3501d45d391bb55398d1620b03edc: Status 404 returned error can't find the container with id e249cbf311605aed3669d7315a5278bd9ab3501d45d391bb55398d1620b03edc Nov 02 20:45:26.220078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.220050 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:45:26.268806 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:26.268774 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:26.327615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327537 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.327615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327584 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.327841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327622 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\" (UniqueName: \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.327841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327648 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.327841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327812 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\" (UniqueName: \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.327994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327846 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:26.327994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327941 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:26.327994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.327952 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\") pod \"1999d0a7-88d0-4686-89d2-591fc33a7e81\" (UID: \"1999d0a7-88d0-4686-89d2-591fc33a7e81\") " Nov 02 20:45:26.328214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.328192 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:26.328484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.328466 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.328484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.328484 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.328642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.328493 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.329988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.329967 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:26.330070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.330020 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:26.330151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.330138 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh" pod "1999d0a7-88d0-4686-89d2-591fc33a7e81" (UID: "1999d0a7-88d0-4686-89d2-591fc33a7e81"). InnerVolumeSpecName "kube-api-access-psnfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:26.425509 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:26.425471 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-9fcacc\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:26.430260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.430222 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psnfh\" (UniqueName: \"kubernetes.io/projected/1999d0a7-88d0-4686-89d2-591fc33a7e81-kube-api-access-psnfh\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.430260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.430263 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999d0a7-88d0-4686-89d2-591fc33a7e81-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.430413 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.430272 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999d0a7-88d0-4686-89d2-591fc33a7e81-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:26.481272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.481245 5136 generic.go:356] "Generic (PLEG): container finished" podID="1999d0a7-88d0-4686-89d2-591fc33a7e81" containerID="2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90" exitCode=0 Nov 02 20:45:26.481418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.481287 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" Nov 02 20:45:26.481418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.481331 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" event={"ID":"1999d0a7-88d0-4686-89d2-591fc33a7e81","Type":"ContainerDied","Data":"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90"} Nov 02 20:45:26.481418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.481373 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv" event={"ID":"1999d0a7-88d0-4686-89d2-591fc33a7e81","Type":"ContainerDied","Data":"c87981b8b6e5c7e2d36af3e2169bd6a2bf8468a3c0af474035206a62a2510522"} Nov 02 20:45:26.481418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.481395 5136 scope.go:122] "RemoveContainer" containerID="2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90" Nov 02 20:45:26.483688 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.483662 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-8d1941" event={"ID":"18412f5a-861b-44fa-9d4b-7878aadc5400","Type":"ContainerStarted","Data":"f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6"} Nov 02 20:45:26.484158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.483703 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-8d1941" event={"ID":"18412f5a-861b-44fa-9d4b-7878aadc5400","Type":"ContainerStarted","Data":"e249cbf311605aed3669d7315a5278bd9ab3501d45d391bb55398d1620b03edc"} Nov 02 20:45:26.492067 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.492048 5136 scope.go:122] "RemoveContainer" containerID="2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90" Nov 02 20:45:26.492334 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:26.492314 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90\": container with ID starting with 2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90 not found: ID does not exist" containerID="2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90" Nov 02 20:45:26.492401 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.492341 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90"} err="failed to get container status \"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90\": rpc error: code = NotFound desc = could not find container \"2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90\": container with ID starting with 2c7ef7d9303680abf04558feb7d323a17f9de881b4f1957792f20b702ce7ec90 not found: ID does not exist" Nov 02 20:45:26.530027 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:26.529997 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:26.547431 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.547398 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" path="/var/lib/kubelet/pods/c73fd09c-b6a6-492f-8a9c-ba3b285c8e43/volumes" Nov 02 20:45:26.627882 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:26.627796 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-9fcacc\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="c73fd09c-b6a6-492f-8a9c-ba3b285c8e43" pod="opendatahub/test-curl-7516-9fcacc" Nov 02 20:45:26.738339 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.738286 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7516-8d1941" podStartSLOduration=1.738271747 podStartE2EDuration="1.738271747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:26.736772504 +0000 UTC m=+3410.343317009" watchObservedRunningTime="2026-11-02 20:45:26.738271747 +0000 UTC m=+3410.344816245" Nov 02 20:45:26.843109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.843005 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:45:26.846174 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.846148 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2c6ce9/auth-test-model-2c6ce9-kserve-75b8b88cb5-twlcv"] Nov 02 20:45:26.849213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.849189 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7519-76f068"] Nov 02 20:45:26.849778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.849764 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1999d0a7-88d0-4686-89d2-591fc33a7e81" containerName="main" Nov 02 20:45:26.849844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.849779 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999d0a7-88d0-4686-89d2-591fc33a7e81" containerName="main" Nov 02 20:45:26.849915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.849905 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999d0a7-88d0-4686-89d2-591fc33a7e81" containerName="main" Nov 02 20:45:26.854439 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.854418 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:26.904103 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.903546 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-76f068"] Nov 02 20:45:26.937014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:26.936977 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679l8\" (UniqueName: \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\") pod \"test-curl-7519-76f068\" (UID: \"56c72e03-5603-463c-9219-3f3cd0e2beaa\") " pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:27.038426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.038394 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679l8\" (UniqueName: \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\") pod \"test-curl-7519-76f068\" (UID: \"56c72e03-5603-463c-9219-3f3cd0e2beaa\") " pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:27.092149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.092117 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-679l8\" (UniqueName: \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\") pod \"test-curl-7519-76f068\" (UID: \"56c72e03-5603-463c-9219-3f3cd0e2beaa\") " pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:27.166581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.166502 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:27.494884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.494860 5136 generic.go:356] "Generic (PLEG): container finished" podID="f0417b32-cae6-4006-8267-fff4faaaa1e6" containerID="be385127316f06c96adad53256db1ae044e339270064a4151f14d139149a7d5f" exitCode=137 Nov 02 20:45:27.533608 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.533580 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7519-76f068"] Nov 02 20:45:27.550578 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:27.550556 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56c72e03_5603_463c_9219_3f3cd0e2beaa.slice/crio-d0e232885893fbe7d14cc44e55f4ecfeb5c3585176b3237228bb4a0207b85137 WatchSource:0}: Error finding container d0e232885893fbe7d14cc44e55f4ecfeb5c3585176b3237228bb4a0207b85137: Status 404 returned error can't find the container with id d0e232885893fbe7d14cc44e55f4ecfeb5c3585176b3237228bb4a0207b85137 Nov 02 20:45:27.679706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.679681 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:27.728074 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:27.728013 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:27.747994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.747972 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\" (UniqueName: \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\") pod \"f0417b32-cae6-4006-8267-fff4faaaa1e6\" (UID: \"f0417b32-cae6-4006-8267-fff4faaaa1e6\") " Nov 02 20:45:27.749996 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.749971 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js" pod "f0417b32-cae6-4006-8267-fff4faaaa1e6" (UID: "f0417b32-cae6-4006-8267-fff4faaaa1e6"). InnerVolumeSpecName "kube-api-access-vg7js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:27.850031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.850004 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7js\" (UniqueName: \"kubernetes.io/projected/f0417b32-cae6-4006-8267-fff4faaaa1e6-kube-api-access-vg7js\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:27.998118 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:27.998037 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-8d1941"] Nov 02 20:45:28.005152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.005123 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-8d1941"] Nov 02 20:45:28.134619 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.134590 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7516-1551d6"] Nov 02 20:45:28.135269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.135252 5136 cpu_manager.go:414] "containerMap: removing container" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" containerName="test-curl-7516-8d1941" Nov 02 20:45:28.135316 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.135271 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" containerName="test-curl-7516-8d1941" Nov 02 20:45:28.135387 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.135378 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" containerName="test-curl-7516-8d1941" Nov 02 20:45:28.138887 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.138868 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:28.153317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.153295 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjmg\" (UniqueName: \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\") pod \"test-curl-7516-1551d6\" (UID: \"2c5a5e7a-9a10-41bf-ade4-5a337524d215\") " pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:28.187202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.187181 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-1551d6"] Nov 02 20:45:28.254264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.254191 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjmg\" (UniqueName: \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\") pod \"test-curl-7516-1551d6\" (UID: \"2c5a5e7a-9a10-41bf-ade4-5a337524d215\") " pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:28.289686 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:28.289658 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:28.307541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.307511 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjmg\" (UniqueName: \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\") pod \"test-curl-7516-1551d6\" (UID: \"2c5a5e7a-9a10-41bf-ade4-5a337524d215\") " pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:28.451206 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.451172 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:28.504974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.504850 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:28.505304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.505284 5136 scope.go:122] "RemoveContainer" containerID="be385127316f06c96adad53256db1ae044e339270064a4151f14d139149a7d5f" Nov 02 20:45:28.508574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.508435 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-76f068" event={"ID":"56c72e03-5603-463c-9219-3f3cd0e2beaa","Type":"ContainerStarted","Data":"b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654"} Nov 02 20:45:28.508574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.508487 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7519-76f068" event={"ID":"56c72e03-5603-463c-9219-3f3cd0e2beaa","Type":"ContainerStarted","Data":"d0e232885893fbe7d14cc44e55f4ecfeb5c3585176b3237228bb4a0207b85137"} Nov 02 20:45:28.508851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.508757 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7516-8d1941" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" containerName="test-curl-7516-8d1941" containerID="cri-o://f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6" gracePeriod=2 Nov 02 20:45:28.552345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.552309 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1999d0a7-88d0-4686-89d2-591fc33a7e81" path="/var/lib/kubelet/pods/1999d0a7-88d0-4686-89d2-591fc33a7e81/volumes" Nov 02 20:45:28.552919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.552899 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" path="/var/lib/kubelet/pods/f0417b32-cae6-4006-8267-fff4faaaa1e6/volumes" Nov 02 20:45:28.553974 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:28.553948 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-c9c6b8\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="f0417b32-cae6-4006-8267-fff4faaaa1e6" pod="opendatahub/test-curl-7516-c9c6b8" Nov 02 20:45:28.661232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.661175 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7519-76f068" podStartSLOduration=2.661158244 podStartE2EDuration="2.661158244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:28.659806597 +0000 UTC m=+3412.266351097" watchObservedRunningTime="2026-11-02 20:45:28.661158244 +0000 UTC m=+3412.267702745" Nov 02 20:45:28.818155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:28.818126 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7516-1551d6"] Nov 02 20:45:28.818267 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:28.818248 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c5a5e7a_9a10_41bf_ade4_5a337524d215.slice/crio-81d31ee7ef4394c78a41c93c2016c003d43af6daa5a4e7d4f7df4eb7bdbef84b WatchSource:0}: Error finding container 81d31ee7ef4394c78a41c93c2016c003d43af6daa5a4e7d4f7df4eb7bdbef84b: Status 404 returned error can't find the container with id 81d31ee7ef4394c78a41c93c2016c003d43af6daa5a4e7d4f7df4eb7bdbef84b Nov 02 20:45:29.517411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:29.517375 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-1551d6" event={"ID":"2c5a5e7a-9a10-41bf-ade4-5a337524d215","Type":"ContainerStarted","Data":"d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4"} Nov 02 20:45:29.517903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:29.517428 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7516-1551d6" event={"ID":"2c5a5e7a-9a10-41bf-ade4-5a337524d215","Type":"ContainerStarted","Data":"81d31ee7ef4394c78a41c93c2016c003d43af6daa5a4e7d4f7df4eb7bdbef84b"} Nov 02 20:45:29.626273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:29.626217 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7516-1551d6" podStartSLOduration=1.626202125 podStartE2EDuration="1.626202125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:29.623959151 +0000 UTC m=+3413.230503648" watchObservedRunningTime="2026-11-02 20:45:29.626202125 +0000 UTC m=+3413.232746627" Nov 02 20:45:29.982538 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:29.982507 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7519-76f068"] Nov 02 20:45:29.986449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:29.986424 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7519-76f068"] Nov 02 20:45:30.524881 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.524718 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7519-76f068" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" containerName="test-curl-7519-76f068" containerID="cri-o://b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654" gracePeriod=2 Nov 02 20:45:30.573238 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:30.573208 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7519-76f068\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:30.599033 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.599003 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerName="main" containerID="cri-o://bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b" gracePeriod=45 Nov 02 20:45:30.694540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.694506 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.157:8000/ready\": dial tcp 10.133.0.157:8000: connect: connection refused" Nov 02 20:45:30.748801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.748771 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:30.784796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.784391 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\" (UniqueName: \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\") pod \"18412f5a-861b-44fa-9d4b-7878aadc5400\" (UID: \"18412f5a-861b-44fa-9d4b-7878aadc5400\") " Nov 02 20:45:30.788235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.788200 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb" pod "18412f5a-861b-44fa-9d4b-7878aadc5400" (UID: "18412f5a-861b-44fa-9d4b-7878aadc5400"). InnerVolumeSpecName "kube-api-access-r9fsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:30.852853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.852829 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:45:30.886512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886487 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886522 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886592 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886630 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886621 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\" (UniqueName: \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886711 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886788 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:30.886833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886797 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\") pod \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\" (UID: \"9412d8d0-946c-4e7d-bf72-b2a32ca505d7\") " Nov 02 20:45:30.886941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.886871 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:30.887084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.887064 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:30.887289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.887271 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.887289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.887291 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.887458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.887307 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9fsb\" (UniqueName: \"kubernetes.io/projected/18412f5a-861b-44fa-9d4b-7878aadc5400-kube-api-access-r9fsb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.887458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.887321 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.888980 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.888961 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:30.889193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.889172 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:30.889193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.889180 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv" pod "9412d8d0-946c-4e7d-bf72-b2a32ca505d7" (UID: "9412d8d0-946c-4e7d-bf72-b2a32ca505d7"). InnerVolumeSpecName "kube-api-access-zxksv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:30.989314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.989276 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxksv\" (UniqueName: \"kubernetes.io/projected/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-kube-api-access-zxksv\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.989314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.989313 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:30.989541 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:30.989330 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9412d8d0-946c-4e7d-bf72-b2a32ca505d7-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:31.024337 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.024306 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7516-1551d6"] Nov 02 20:45:31.028017 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.027994 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7516-1551d6"] Nov 02 20:45:31.533420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.533384 5136 generic.go:356] "Generic (PLEG): container finished" podID="18412f5a-861b-44fa-9d4b-7878aadc5400" containerID="f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6" exitCode=137 Nov 02 20:45:31.533799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.533444 5136 scope.go:122] "RemoveContainer" containerID="f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6" Nov 02 20:45:31.533799 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.533448 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:31.535128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.535107 5136 generic.go:356] "Generic (PLEG): container finished" podID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerID="bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b" exitCode=0 Nov 02 20:45:31.535228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.535167 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" Nov 02 20:45:31.535228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.535185 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" event={"ID":"9412d8d0-946c-4e7d-bf72-b2a32ca505d7","Type":"ContainerDied","Data":"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b"} Nov 02 20:45:31.535332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.535231 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh" event={"ID":"9412d8d0-946c-4e7d-bf72-b2a32ca505d7","Type":"ContainerDied","Data":"8545a68bb38d8a709eed7fe2ebcbfc150631efffa322f64c8af19b8ec5742d91"} Nov 02 20:45:31.535414 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.535396 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7516-1551d6" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" containerName="test-curl-7516-1551d6" containerID="cri-o://d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4" gracePeriod=2 Nov 02 20:45:31.543561 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.543538 5136 scope.go:122] "RemoveContainer" containerID="f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6" Nov 02 20:45:31.543851 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:31.543827 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6\": container with ID starting with f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6 not found: ID does not exist" containerID="f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6" Nov 02 20:45:31.543943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.543859 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6"} err="failed to get container status \"f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6\": rpc error: code = NotFound desc = could not find container \"f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6\": container with ID starting with f76039d57f20d4650d533a24303babf1fb01f212e322a6ff67a6113ba5bbc5a6 not found: ID does not exist" Nov 02 20:45:31.543943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.543883 5136 scope.go:122] "RemoveContainer" containerID="bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b" Nov 02 20:45:31.552657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.552641 5136 scope.go:122] "RemoveContainer" containerID="bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b" Nov 02 20:45:31.553522 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:31.553506 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b\": container with ID starting with bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b not found: ID does not exist" containerID="bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b" Nov 02 20:45:31.553588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.553527 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b"} err="failed to get container status \"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b\": rpc error: code = NotFound desc = could not find container \"bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b\": container with ID starting with bcca1f557e718ca38caa2c2c8cab919a35be30bb502ca1b7ca6087d6e176b68b not found: ID does not exist" Nov 02 20:45:31.633270 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:31.633237 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-8d1941\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:31.681552 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:31.681517 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-1551d6\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:31.788207 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.788127 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:45:31.790629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.790601 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-78bef7/auth-test-model-78bef7-kserve-58b86dfdd5-9s8zh"] Nov 02 20:45:31.979756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.979696 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:45:31.980321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980306 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerName="main" Nov 02 20:45:31.980369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980322 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerName="main" Nov 02 20:45:31.980369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980331 5136 cpu_manager.go:414] "containerMap: removing container" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" containerName="test-curl-7519-76f068" Nov 02 20:45:31.980369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980337 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" containerName="test-curl-7519-76f068" Nov 02 20:45:31.980369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980349 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" containerName="test-curl-7516-1551d6" Nov 02 20:45:31.980369 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980353 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" containerName="test-curl-7516-1551d6" Nov 02 20:45:31.980527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980449 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" containerName="main" Nov 02 20:45:31.980527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980457 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" containerName="test-curl-7519-76f068" Nov 02 20:45:31.980527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.980466 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" containerName="test-curl-7516-1551d6" Nov 02 20:45:31.985300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:31.985280 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.040850 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:32.040140 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-8d1941\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" pod="opendatahub/test-curl-7516-8d1941" Nov 02 20:45:32.070118 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.070079 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0a8f62-openshift-default-dockercfg-sxkn8\"" Nov 02 20:45:32.072573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.072545 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:45:32.103175 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103146 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q98p\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103196 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103267 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103384 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103315 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103562 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103533 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103615 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103580 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103719 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103701 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103760 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.103879 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.103850 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205296 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205261 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q98p\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205476 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205314 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205481 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205516 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205672 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205653 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205690 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205778 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205838 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205817 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205866 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.205938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205918 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.206012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.205988 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.206301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.206270 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.206301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.206287 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.206499 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.206457 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.207870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.207842 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.207974 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:32.207865 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7516-1551d6\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:32.207974 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.207942 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.260465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.260434 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.260579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.260502 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q98p\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\") pod \"e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.299160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.299105 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:32.547993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.547968 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18412f5a-861b-44fa-9d4b-7878aadc5400" path="/var/lib/kubelet/pods/18412f5a-861b-44fa-9d4b-7878aadc5400/volumes" Nov 02 20:45:32.548438 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.548423 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9412d8d0-946c-4e7d-bf72-b2a32ca505d7" path="/var/lib/kubelet/pods/9412d8d0-946c-4e7d-bf72-b2a32ca505d7/volumes" Nov 02 20:45:32.722995 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.722965 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:45:32.724403 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:32.724379 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bcf0f1_b043_4239_b2de_8766ebbe7072.slice/crio-22a4b9b1d84b2d8959e2c0e7a7db52a89aa55da60d6e9236ae5f45186114cf9d WatchSource:0}: Error finding container 22a4b9b1d84b2d8959e2c0e7a7db52a89aa55da60d6e9236ae5f45186114cf9d: Status 404 returned error can't find the container with id 22a4b9b1d84b2d8959e2c0e7a7db52a89aa55da60d6e9236ae5f45186114cf9d Nov 02 20:45:32.726785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.726743 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:32.726862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.726835 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:32.726906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.726878 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:32.796346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.796320 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:32.914649 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.914568 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\" (UniqueName: \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\") pod \"56c72e03-5603-463c-9219-3f3cd0e2beaa\" (UID: \"56c72e03-5603-463c-9219-3f3cd0e2beaa\") " Nov 02 20:45:32.916753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:32.916694 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8" pod "56c72e03-5603-463c-9219-3f3cd0e2beaa" (UID: "56c72e03-5603-463c-9219-3f3cd0e2beaa"). InnerVolumeSpecName "kube-api-access-679l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:33.021104 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.020978 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-679l8\" (UniqueName: \"kubernetes.io/projected/56c72e03-5603-463c-9219-3f3cd0e2beaa-kube-api-access-679l8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:33.126358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.126326 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:45:33.130778 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.130751 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.186169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.185658 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223606 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223680 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223768 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbccm\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223806 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223843 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223910 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.223932 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.224041 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.225147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.224073 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.235498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.235384 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8d5148-openshift-default-dockercfg-swxjz\"" Nov 02 20:45:33.325190 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325156 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325222 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325268 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbccm\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325302 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325334 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325399 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325428 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325531 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325565 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325577 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.325939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.325900 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.326024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.326000 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.326324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.326292 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.326432 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.326414 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.327805 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.327783 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.328248 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.328226 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.379772 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.379722 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.379926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.379879 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbccm\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\") pod \"e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.448283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.448261 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:33.588078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.587613 5136 generic.go:356] "Generic (PLEG): container finished" podID="56c72e03-5603-463c-9219-3f3cd0e2beaa" containerID="b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654" exitCode=137 Nov 02 20:45:33.588078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.587811 5136 scope.go:122] "RemoveContainer" containerID="b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654" Nov 02 20:45:33.588078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.587939 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:33.593464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.593125 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerStarted","Data":"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128"} Nov 02 20:45:33.593464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.593179 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerStarted","Data":"22a4b9b1d84b2d8959e2c0e7a7db52a89aa55da60d6e9236ae5f45186114cf9d"} Nov 02 20:45:33.658668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.658644 5136 scope.go:122] "RemoveContainer" containerID="b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654" Nov 02 20:45:33.659037 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:33.659018 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654\": container with ID starting with b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654 not found: ID does not exist" containerID="b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654" Nov 02 20:45:33.659084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.659046 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654"} err="failed to get container status \"b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654\": rpc error: code = NotFound desc = could not find container \"b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654\": container with ID starting with b95c8141a4ffff47e77e59590dc26219f0ae6844b1ad49f06125a3b58ce22654 not found: ID does not exist" Nov 02 20:45:33.708609 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.708083 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podStartSLOduration=2.708064827 podStartE2EDuration="2.708064827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:33.704717453 +0000 UTC m=+3417.311261953" watchObservedRunningTime="2026-11-02 20:45:33.708064827 +0000 UTC m=+3417.314609332" Nov 02 20:45:33.753081 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:33.753048 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7519-76f068\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:33.786083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.786059 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:33.839528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.832382 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\" (UniqueName: \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\") pod \"2c5a5e7a-9a10-41bf-ade4-5a337524d215\" (UID: \"2c5a5e7a-9a10-41bf-ade4-5a337524d215\") " Nov 02 20:45:33.839528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.837040 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg" pod "2c5a5e7a-9a10-41bf-ade4-5a337524d215" (UID: "2c5a5e7a-9a10-41bf-ade4-5a337524d215"). InnerVolumeSpecName "kube-api-access-tjjmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:33.839528 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:33.837067 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7519-76f068\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:33.871898 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.871871 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:45:33.873593 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:33.873567 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f3fd04_2c44_41f8_9796_7f704a9b466b.slice/crio-318643183e80abfeb7345cb70513df5413b5c77a8798000088a6aecedda393bb WatchSource:0}: Error finding container 318643183e80abfeb7345cb70513df5413b5c77a8798000088a6aecedda393bb: Status 404 returned error can't find the container with id 318643183e80abfeb7345cb70513df5413b5c77a8798000088a6aecedda393bb Nov 02 20:45:33.876335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.876301 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:33.876406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.876370 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:33.876406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.876399 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:33.934788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:33.934766 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjmg\" (UniqueName: \"kubernetes.io/projected/2c5a5e7a-9a10-41bf-ade4-5a337524d215-kube-api-access-tjjmg\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:34.299509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.299482 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:34.305039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.305016 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:34.353720 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:34.353688 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7519-76f068\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" pod="opendatahub/test-curl-7519-76f068" Nov 02 20:45:34.551787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.551688 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" path="/var/lib/kubelet/pods/2c5a5e7a-9a10-41bf-ade4-5a337524d215/volumes" Nov 02 20:45:34.554088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.552326 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c72e03-5603-463c-9219-3f3cd0e2beaa" path="/var/lib/kubelet/pods/56c72e03-5603-463c-9219-3f3cd0e2beaa/volumes" Nov 02 20:45:34.600042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.600015 5136 generic.go:356] "Generic (PLEG): container finished" podID="2c5a5e7a-9a10-41bf-ade4-5a337524d215" containerID="d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4" exitCode=137 Nov 02 20:45:34.600473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.600064 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7516-1551d6" Nov 02 20:45:34.600473 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.600107 5136 scope.go:122] "RemoveContainer" containerID="d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4" Nov 02 20:45:34.602275 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.602026 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerStarted","Data":"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb"} Nov 02 20:45:34.602275 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.602085 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerStarted","Data":"318643183e80abfeb7345cb70513df5413b5c77a8798000088a6aecedda393bb"} Nov 02 20:45:34.603520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.603502 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:34.604554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.604534 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:45:34.611956 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.611937 5136 scope.go:122] "RemoveContainer" containerID="d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4" Nov 02 20:45:34.612387 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:34.612367 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4\": container with ID starting with d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4 not found: ID does not exist" containerID="d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4" Nov 02 20:45:34.612460 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.612398 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4"} err="failed to get container status \"d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4\": rpc error: code = NotFound desc = could not find container \"d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4\": container with ID starting with d50ddbcf9b15e7349f78c68cc105cd630f2e005ee896ef102e49f1d0c46a75c4 not found: ID does not exist" Nov 02 20:45:34.714310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:34.714258 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podStartSLOduration=1.7142406079999999 podStartE2EDuration="1.714240608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:34.712663846 +0000 UTC m=+3418.319208345" watchObservedRunningTime="2026-11-02 20:45:34.714240608 +0000 UTC m=+3418.320785108" Nov 02 20:45:35.449169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:35.449142 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:35.454835 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:35.454809 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:35.611085 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:35.611055 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:35.611998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:35.611981 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:45:44.584970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.584937 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:45:44.589112 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.589090 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.639211 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.639182 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8d5148-dockercfg-wcbm2\"" Nov 02 20:45:44.642415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.642385 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:45:44.700782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.698112 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8d5148\"" Nov 02 20:45:44.714892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.714861 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks9v\" (UniqueName: \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.716238 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.716212 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.818018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.817958 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lks9v\" (UniqueName: \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.818220 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.818077 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.818982 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.818953 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.872459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.872396 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks9v\" (UniqueName: \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\") pod \"payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:44.903228 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:44.903205 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:45.270030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:45.270002 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:45:45.271051 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:45:45.271017 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825b8950_821e_4778_8c18_840bdc5e11f8.slice/crio-e2299b8685711d5af25f2923727e14ea7c668e0fe717b056cda86ef0ffb4fc82 WatchSource:0}: Error finding container e2299b8685711d5af25f2923727e14ea7c668e0fe717b056cda86ef0ffb4fc82: Status 404 returned error can't find the container with id e2299b8685711d5af25f2923727e14ea7c668e0fe717b056cda86ef0ffb4fc82 Nov 02 20:45:45.741120 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:45.741088 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" event={"ID":"825b8950-821e-4778-8c18-840bdc5e11f8","Type":"ContainerStarted","Data":"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25"} Nov 02 20:45:45.741568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:45.741142 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" event={"ID":"825b8950-821e-4778-8c18-840bdc5e11f8","Type":"ContainerStarted","Data":"e2299b8685711d5af25f2923727e14ea7c668e0fe717b056cda86ef0ffb4fc82"} Nov 02 20:45:45.741568 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:45.741171 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:45:45.850648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:45.850590 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" podStartSLOduration=1.850573578 podStartE2EDuration="1.850573578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:45.848656644 +0000 UTC m=+3429.455201141" watchObservedRunningTime="2026-11-02 20:45:45.850573578 +0000 UTC m=+3429.457118137" Nov 02 20:45:51.004300 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:45:51.004225 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:45:56.751355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:45:56.751325 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:46:02.225038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.224995 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7510-5998f7"] Nov 02 20:46:02.229453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.229433 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:02.279335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.279311 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7510-5998f7"] Nov 02 20:46:02.323570 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.323540 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\"" Nov 02 20:46:02.326584 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.326560 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8vg\" (UniqueName: \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\") pod \"test-curl-7510-5998f7\" (UID: \"acee5545-a9ff-47ec-ab28-1c7af576f73f\") " pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:02.427785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.427712 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8vg\" (UniqueName: \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\") pod \"test-curl-7510-5998f7\" (UID: \"acee5545-a9ff-47ec-ab28-1c7af576f73f\") " pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:02.481911 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.481845 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8vg\" (UniqueName: \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\") pod \"test-curl-7510-5998f7\" (UID: \"acee5545-a9ff-47ec-ab28-1c7af576f73f\") " pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:02.541632 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.541593 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:02.907707 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:02.907678 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7510-5998f7"] Nov 02 20:46:02.908693 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:46:02.908653 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacee5545_a9ff_47ec_ab28_1c7af576f73f.slice/crio-d41b709c05f1573bc3de1eb16a7c756b66e520b38e89d267191082add69b65a9 WatchSource:0}: Error finding container d41b709c05f1573bc3de1eb16a7c756b66e520b38e89d267191082add69b65a9: Status 404 returned error can't find the container with id d41b709c05f1573bc3de1eb16a7c756b66e520b38e89d267191082add69b65a9 Nov 02 20:46:03.860019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:03.859986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7510-5998f7" event={"ID":"acee5545-a9ff-47ec-ab28-1c7af576f73f","Type":"ContainerStarted","Data":"8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786"} Nov 02 20:46:03.860019 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:03.860025 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7510-5998f7" event={"ID":"acee5545-a9ff-47ec-ab28-1c7af576f73f","Type":"ContainerStarted","Data":"d41b709c05f1573bc3de1eb16a7c756b66e520b38e89d267191082add69b65a9"} Nov 02 20:46:03.971852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:03.971786 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7510-5998f7" podStartSLOduration=1.9717646530000001 podStartE2EDuration="1.971764653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:03.970896623 +0000 UTC m=+3447.577441122" watchObservedRunningTime="2026-11-02 20:46:03.971764653 +0000 UTC m=+3447.578309151" Nov 02 20:46:05.382317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:05.382285 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7510-5998f7"] Nov 02 20:46:05.388036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:05.388005 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7510-5998f7"] Nov 02 20:46:05.871194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:05.871153 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7510-5998f7" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" containerName="test-curl-7510-5998f7" containerID="cri-o://8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786" gracePeriod=2 Nov 02 20:46:05.919807 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:05.919772 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7510-5998f7\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:06.590745 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:06.590700 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7510-5998f7\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:08.112598 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.112573 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:08.197797 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.197764 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\" (UniqueName: \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\") pod \"acee5545-a9ff-47ec-ab28-1c7af576f73f\" (UID: \"acee5545-a9ff-47ec-ab28-1c7af576f73f\") " Nov 02 20:46:08.200083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.200051 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg" pod "acee5545-a9ff-47ec-ab28-1c7af576f73f" (UID: "acee5545-a9ff-47ec-ab28-1c7af576f73f"). InnerVolumeSpecName "kube-api-access-zx8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:08.299844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.299811 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8vg\" (UniqueName: \"kubernetes.io/projected/acee5545-a9ff-47ec-ab28-1c7af576f73f-kube-api-access-zx8vg\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:08.547498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.547422 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" path="/var/lib/kubelet/pods/acee5545-a9ff-47ec-ab28-1c7af576f73f/volumes" Nov 02 20:46:08.891851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.891767 5136 generic.go:356] "Generic (PLEG): container finished" podID="acee5545-a9ff-47ec-ab28-1c7af576f73f" containerID="8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786" exitCode=137 Nov 02 20:46:08.891851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.891805 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7510-5998f7" Nov 02 20:46:08.891851 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.891838 5136 scope.go:122] "RemoveContainer" containerID="8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786" Nov 02 20:46:08.900912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.900887 5136 scope.go:122] "RemoveContainer" containerID="8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786" Nov 02 20:46:08.901198 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:08.901174 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786\": container with ID starting with 8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786 not found: ID does not exist" containerID="8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786" Nov 02 20:46:08.901280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:08.901208 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786"} err="failed to get container status \"8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786\": rpc error: code = NotFound desc = could not find container \"8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786\": container with ID starting with 8dbd7be64a30010ef7037ec331a2b5a926074d8c07d4367248c084c67fe1f786 not found: ID does not exist" Nov 02 20:46:21.003075 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:21.003032 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:46:30.074444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:30.074413 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:46:45.095388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.095346 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerName="main" containerID="cri-o://f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b" gracePeriod=45 Nov 02 20:46:45.138399 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.138362 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused" Nov 02 20:46:45.361041 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.360980 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:46:45.497707 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497677 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.497890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497716 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.497890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497765 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.497890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497819 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.498082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497900 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\" (UniqueName: \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.498082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497927 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\") pod \"9ba40a86-c504-4f76-81f8-f0d9048715ba\" (UID: \"9ba40a86-c504-4f76-81f8-f0d9048715ba\") " Nov 02 20:46:45.498082 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.497962 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.498258 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.498152 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.498472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.498443 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.498845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.498708 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.499005 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.498981 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.499125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.499008 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.500452 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.500421 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:45.500610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.500573 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "kube-api-access-kdfq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:45.500783 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.500760 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm" pod "9ba40a86-c504-4f76-81f8-f0d9048715ba" (UID: "9ba40a86-c504-4f76-81f8-f0d9048715ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.600695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.600659 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba40a86-c504-4f76-81f8-f0d9048715ba-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.600695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.600685 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba40a86-c504-4f76-81f8-f0d9048715ba-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.600695 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:45.600696 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfq8\" (UniqueName: \"kubernetes.io/projected/9ba40a86-c504-4f76-81f8-f0d9048715ba-kube-api-access-kdfq8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.121332 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.121294 5136 generic.go:356] "Generic (PLEG): container finished" podID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerID="f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b" exitCode=0 Nov 02 20:46:46.121820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.121362 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" Nov 02 20:46:46.121820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.121380 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" event={"ID":"9ba40a86-c504-4f76-81f8-f0d9048715ba","Type":"ContainerDied","Data":"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b"} Nov 02 20:46:46.121820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.121429 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs" event={"ID":"9ba40a86-c504-4f76-81f8-f0d9048715ba","Type":"ContainerDied","Data":"4519de80c46091a8b02faf591db6dd019043beee0cdeb83b26470a75124fdb7e"} Nov 02 20:46:46.121820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.121454 5136 scope.go:122] "RemoveContainer" containerID="f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b" Nov 02 20:46:46.131903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.131885 5136 scope.go:122] "RemoveContainer" containerID="f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b" Nov 02 20:46:46.132181 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:46.132164 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b\": container with ID starting with f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b not found: ID does not exist" containerID="f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b" Nov 02 20:46:46.132222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.132189 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b"} err="failed to get container status \"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b\": rpc error: code = NotFound desc = could not find container \"f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b\": container with ID starting with f412eb70b26ba8393c3f52529240c122642283f481f51132877254076789186b not found: ID does not exist" Nov 02 20:46:46.276981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.276951 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:46:46.324266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.324232 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-44a484/auth-test-model-44a484-kserve-949f95889-66jvs"] Nov 02 20:46:46.554706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:46.554672 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" path="/var/lib/kubelet/pods/9ba40a86-c504-4f76-81f8-f0d9048715ba/volumes" Nov 02 20:46:47.757718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.757684 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:46:47.758304 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758290 5136 cpu_manager.go:414] "containerMap: removing container" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" containerName="test-curl-7510-5998f7" Nov 02 20:46:47.758346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758305 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" containerName="test-curl-7510-5998f7" Nov 02 20:46:47.758346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758345 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerName="main" Nov 02 20:46:47.758420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758351 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerName="main" Nov 02 20:46:47.758458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758443 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba40a86-c504-4f76-81f8-f0d9048715ba" containerName="main" Nov 02 20:46:47.758458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.758456 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="acee5545-a9ff-47ec-ab28-1c7af576f73f" containerName="test-curl-7510-5998f7" Nov 02 20:46:47.763232 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.763211 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:47.814271 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.814232 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:46:47.857938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.857912 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0a8f62-serving-cert\"" Nov 02 20:46:47.930651 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.930614 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:47.930820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.930756 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:47.930876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.930823 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzf5\" (UniqueName: \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:47.930941 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:47.930917 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.032317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.032228 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.032317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.032294 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzf5\" (UniqueName: \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.032566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.032365 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.032566 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.032431 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.032566 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:48.032538 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-serving-cert: secret "maas-api-e2e-extmod-a-0a8f62-serving-cert" not found Nov 02 20:46:48.032754 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:48.032620 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls podName:47fa2c93-dc2a-4cde-9d63-1b8e31388366 nodeName:}" failed. No retries permitted until 2026-11-02 20:46:48.532598526 +0000 UTC m=+3492.139143023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls") pod "maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" (UID: "47fa2c93-dc2a-4cde-9d63-1b8e31388366") : secret "maas-api-e2e-extmod-a-0a8f62-serving-cert" not found Nov 02 20:46:48.033128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.033105 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.034882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.034860 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.087100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.087076 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzf5\" (UniqueName: \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.539360 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.539322 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.541788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.541765 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:48.676957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:48.676931 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:49.045225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:49.045203 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:46:49.046694 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:46:49.046663 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fa2c93_dc2a_4cde_9d63_1b8e31388366.slice/crio-7ac413bf343ebb8ec48f47a4d22f8bb7ab8f84e9a06cb5715bafa9d7ec3fde84 WatchSource:0}: Error finding container 7ac413bf343ebb8ec48f47a4d22f8bb7ab8f84e9a06cb5715bafa9d7ec3fde84: Status 404 returned error can't find the container with id 7ac413bf343ebb8ec48f47a4d22f8bb7ab8f84e9a06cb5715bafa9d7ec3fde84 Nov 02 20:46:49.048347 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:49.048265 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:46:49.142094 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:49.142061 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" event={"ID":"47fa2c93-dc2a-4cde-9d63-1b8e31388366","Type":"ContainerStarted","Data":"7ac413bf343ebb8ec48f47a4d22f8bb7ab8f84e9a06cb5715bafa9d7ec3fde84"} Nov 02 20:46:50.148916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:50.148883 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" event={"ID":"47fa2c93-dc2a-4cde-9d63-1b8e31388366","Type":"ContainerStarted","Data":"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde"} Nov 02 20:46:50.149303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:50.148954 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:50.298749 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:50.298632 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" podStartSLOduration=2.319261558 podStartE2EDuration="3.298616724s" totalImagesPullingTime="979.355166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:50.298213944 +0000 UTC m=+3493.904758443" watchObservedRunningTime="2026-11-02 20:46:50.298616724 +0000 UTC m=+3493.905161221" Nov 02 20:46:51.004179 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:51.004141 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:46:52.639273 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.639233 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:46:52.643821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.643798 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.696803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.696777 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:46:52.740788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.740757 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8d5148-serving-cert\"" Nov 02 20:46:52.797061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.797027 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hzx\" (UniqueName: \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.797310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.797286 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.797512 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.797491 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.797683 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.797659 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.898945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.898871 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.898945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.898908 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.898945 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.898925 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.899222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.898959 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hzx\" (UniqueName: \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.899669 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.899649 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.901378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.901355 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.901477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.901410 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.952960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.952932 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hzx\" (UniqueName: \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\") pod \"maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:52.956899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:52.956879 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:53.282355 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:53.282326 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:46:53.282560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:53.282510 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" containerID="cri-o://55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e" gracePeriod=30 Nov 02 20:46:53.326922 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:53.325436 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:46:53.329305 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:46:53.329270 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431fe126_f76f_4149_8815_c86e19275ae9.slice/crio-1e69c98d97cac3e2a6e6dd212cd5280a1990ca657659b83ba4d9e75b76e69134 WatchSource:0}: Error finding container 1e69c98d97cac3e2a6e6dd212cd5280a1990ca657659b83ba4d9e75b76e69134: Status 404 returned error can't find the container with id 1e69c98d97cac3e2a6e6dd212cd5280a1990ca657659b83ba4d9e75b76e69134 Nov 02 20:46:54.177629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:54.177594 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" event={"ID":"431fe126-f76f-4149-8815-c86e19275ae9","Type":"ContainerStarted","Data":"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f"} Nov 02 20:46:54.178077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:54.177652 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" event={"ID":"431fe126-f76f-4149-8815-c86e19275ae9","Type":"ContainerStarted","Data":"1e69c98d97cac3e2a6e6dd212cd5280a1990ca657659b83ba4d9e75b76e69134"} Nov 02 20:46:54.178077 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:54.177675 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:46:54.288123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:54.288064 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" podStartSLOduration=1.531960253 podStartE2EDuration="2.288047359s" totalImagesPullingTime="756.087106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:54.286197733 +0000 UTC m=+3497.892742231" watchObservedRunningTime="2026-11-02 20:46:54.288047359 +0000 UTC m=+3497.894591859" Nov 02 20:46:56.163091 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:56.163064 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:46:56.581910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:56.581871 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:46:56.582096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:56.581937 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:46:58.559462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.559399 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:46:58.672490 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672460 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672511 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672563 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672586 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672630 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672664 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.672875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672726 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:58.672875 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672858 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.673015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672921 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.673015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672953 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:58.673015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.672974 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\") pod \"22f47f1f-f6eb-4500-b510-bd3125a278ce\" (UID: \"22f47f1f-f6eb-4500-b510-bd3125a278ce\") " Nov 02 20:46:58.673015 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673002 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:58.673258 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673209 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:58.673354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673336 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:58.673527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673508 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.673597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673528 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.673597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673537 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.673597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673546 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f47f1f-f6eb-4500-b510-bd3125a278ce-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.673597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.673555 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.675300 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.675281 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:58.675397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.675313 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "kube-api-access-lnv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:58.675397 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.675379 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:58.675504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.675452 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy" pod "22f47f1f-f6eb-4500-b510-bd3125a278ce" (UID: "22f47f1f-f6eb-4500-b510-bd3125a278ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:58.775239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.775205 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnv6z\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-kube-api-access-lnv6z\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.775239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.775233 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.775239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.775244 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:58.775462 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:58.775253 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f47f1f-f6eb-4500-b510-bd3125a278ce-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.212259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.212224 5136 generic.go:356] "Generic (PLEG): container finished" podID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerID="55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e" exitCode=0 Nov 02 20:46:59.212442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.212273 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" Nov 02 20:46:59.212442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.212311 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" event={"ID":"22f47f1f-f6eb-4500-b510-bd3125a278ce","Type":"ContainerDied","Data":"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e"} Nov 02 20:46:59.212442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.212366 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x" event={"ID":"22f47f1f-f6eb-4500-b510-bd3125a278ce","Type":"ContainerDied","Data":"312debf8c6b9b98748b44dd8311640e825709ba28400d6d87c5982eabdee6355"} Nov 02 20:46:59.212442 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.212390 5136 scope.go:122] "RemoveContainer" containerID="55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e" Nov 02 20:46:59.223700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.223682 5136 scope.go:122] "RemoveContainer" containerID="55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e" Nov 02 20:46:59.223992 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:46:59.223975 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e\": container with ID starting with 55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e not found: ID does not exist" containerID="55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e" Nov 02 20:46:59.224050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.224000 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e"} err="failed to get container status \"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e\": rpc error: code = NotFound desc = could not find container \"55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e\": container with ID starting with 55febd6b53fd46df49ae1f59b8b2194070cff4bd4c022878613c7cd912431b0e not found: ID does not exist" Nov 02 20:46:59.369124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.369090 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:46:59.372752 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:46:59.372715 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-ttx2x"] Nov 02 20:47:00.188925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:00.188897 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:47:00.547030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:00.547002 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" path="/var/lib/kubelet/pods/22f47f1f-f6eb-4500-b510-bd3125a278ce/volumes" Nov 02 20:47:02.122009 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.121973 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:47:02.122577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.122563 5136 cpu_manager.go:414] "containerMap: removing container" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" Nov 02 20:47:02.122633 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.122577 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" Nov 02 20:47:02.122718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.122708 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f47f1f-f6eb-4500-b510-bd3125a278ce" containerName="istio-proxy" Nov 02 20:47:02.127852 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.127830 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.217554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217517 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.217765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217619 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.217765 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217667 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.217894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217876 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.217952 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217925 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.218016 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.217956 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.218097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.218074 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.218155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.218143 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.218242 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.218226 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74v6\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.220284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.220258 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:47:02.264193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.264159 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a7c5d1-openshift-default-dockercfg-x24w9\"" Nov 02 20:47:02.319354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319300 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319354 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319366 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319389 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319441 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319478 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319506 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319627 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319565 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319904 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319633 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.319904 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319694 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74v6\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.320020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.319970 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.320020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.320008 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.320143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.320096 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.320264 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.320151 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.320407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.320387 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.321895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.321875 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.322255 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.322233 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.377010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.376930 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.377150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.377003 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74v6\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\") pod \"e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.442222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.442188 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:02.889820 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.888513 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:47:02.912038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.910991 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:02.912038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.911092 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:02.912038 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:02.911138 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:03.242596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:03.242562 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerStarted","Data":"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5"} Nov 02 20:47:03.243130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:03.242617 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerStarted","Data":"6138890b4273f24e7e860e640a40d0d3b8dac65aa68b39f48233a999071d1f5a"} Nov 02 20:47:03.356524 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:03.356465 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podStartSLOduration=1.35644934 podStartE2EDuration="1.35644934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:03.354619744 +0000 UTC m=+3506.961164242" watchObservedRunningTime="2026-11-02 20:47:03.35644934 +0000 UTC m=+3506.962993837" Nov 02 20:47:03.442616 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:03.442584 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:04.448726 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:04.448697 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:05.255282 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:05.255249 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:05.256299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:05.256279 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:47:11.906742 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:11.906695 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:47:11.910929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:11.910909 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:11.963216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:11.963189 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:47:12.013832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.010135 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a7c5d1\"" Nov 02 20:47:12.013832 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.010246 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a7c5d1-dockercfg-nj7sm\"" Nov 02 20:47:12.051646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.051605 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.051900 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.051669 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjtw\" (UniqueName: \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.153212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.153174 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.153418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.153247 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjtw\" (UniqueName: \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.153916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.153898 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.207018 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.206991 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjtw\" (UniqueName: \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\") pod \"payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.224620 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.224590 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:12.593581 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:12.593546 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:47:12.594690 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:47:12.594663 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29f43a7_26a4_4d7f_9c9b_7afcf68051c4.slice/crio-9bc6493e0d52aea9f4076dd3755a50db442a2b837a8372e5ff0c42e77b5c3c55 WatchSource:0}: Error finding container 9bc6493e0d52aea9f4076dd3755a50db442a2b837a8372e5ff0c42e77b5c3c55: Status 404 returned error can't find the container with id 9bc6493e0d52aea9f4076dd3755a50db442a2b837a8372e5ff0c42e77b5c3c55 Nov 02 20:47:13.327287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:13.327214 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" event={"ID":"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4","Type":"ContainerStarted","Data":"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2"} Nov 02 20:47:13.327287 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:13.327282 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" event={"ID":"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4","Type":"ContainerStarted","Data":"9bc6493e0d52aea9f4076dd3755a50db442a2b837a8372e5ff0c42e77b5c3c55"} Nov 02 20:47:13.328410 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:13.328369 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:13.436163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:13.436108 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" podStartSLOduration=2.436085779 podStartE2EDuration="2.436085779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:13.435156146 +0000 UTC m=+3517.041700646" watchObservedRunningTime="2026-11-02 20:47:13.436085779 +0000 UTC m=+3517.042630277" Nov 02 20:47:21.003426 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:47:21.003346 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:47:25.341915 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:25.341883 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:47:51.017249 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:47:51.017202 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:47:54.651586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.651522 5136 generic.go:356] "Generic (PLEG): container finished" podID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerID="bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128" exitCode=0 Nov 02 20:47:54.651586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.651554 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerDied","Data":"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128"} Nov 02 20:47:54.652165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.651952 5136 scope.go:122] "RemoveContainer" containerID="bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128" Nov 02 20:47:54.653385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653368 5136 generic.go:356] "Generic (PLEG): container finished" podID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerID="6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb" exitCode=0 Nov 02 20:47:54.653458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653438 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerDied","Data":"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb"} Nov 02 20:47:54.653549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653522 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:54.653601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653588 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:54.653648 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653623 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:54.653763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.653724 5136 scope.go:122] "RemoveContainer" containerID="6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb" Nov 02 20:47:54.655111 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.655086 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:54.655185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.655147 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:54.655185 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:54.655178 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:55.662083 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:55.662040 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerStarted","Data":"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915"} Nov 02 20:47:55.664326 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:55.664301 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerStarted","Data":"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94"} Nov 02 20:47:56.309626 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.309596 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:47:56.310040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.309945 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:47:56.310215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.310193 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 20:47:56.449093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.449052 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:47:56.449491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.449467 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:47:56.449617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:56.449527 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 20:47:57.306756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:57.305723 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:47:57.306756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:57.305812 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 20:47:57.449303 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:57.449259 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:47:57.449481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:57.449318 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 20:47:58.307890 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:58.307847 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:47:58.308535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:58.308502 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 20:47:58.449215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:58.449179 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:47:58.449472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:58.449441 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 20:47:59.304188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:59.304145 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:47:59.305450 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:59.305418 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 20:47:59.448965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:59.448936 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:47:59.449411 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:47:59.449387 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 20:48:00.304212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:00.304174 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:48:00.304519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:00.304491 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 02 20:48:00.449528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:00.449490 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:48:00.449958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:00.449541 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 20:48:01.304059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.304030 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:48:01.304351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.304331 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:48:01.305137 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.305120 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:48:01.453647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.453615 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:48:01.454212 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.454193 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:48:01.454929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:01.454908 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:48:21.004781 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:48:21.004715 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:48:22.591428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.591394 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:48:22.594813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.594791 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.644368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.644331 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-27e421-openshift-default-dockercfg-pkh66\"" Nov 02 20:48:22.693380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693345 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693398 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693496 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693524 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693762 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693824 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.693912 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693861 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.694007 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.693983 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.694169 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.694150 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqtt\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.695278 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.695244 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798551 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798604 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798636 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798760 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798807 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqtt\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798858 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798887 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798922 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.798976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.798944 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.799635 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.799475 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.832811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.800185 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.832811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.800849 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.832811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.829254 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.832811 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.829904 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.835087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.833380 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.835965 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.835941 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.856026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.855948 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.883266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.883233 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqtt\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\") pod \"e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:22.907509 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:22.907485 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:23.323931 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.323897 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:48:23.324836 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:48:23.324809 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec6eae_4d68_4630_9b47_3d5826c94192.slice/crio-86a1bf6d110d570094cb97af11c6de2ba8c2d52b68f96a157d9106164def3f3b WatchSource:0}: Error finding container 86a1bf6d110d570094cb97af11c6de2ba8c2d52b68f96a157d9106164def3f3b: Status 404 returned error can't find the container with id 86a1bf6d110d570094cb97af11c6de2ba8c2d52b68f96a157d9106164def3f3b Nov 02 20:48:23.327725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.327693 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:48:23.327831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.327775 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:48:23.327831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.327808 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:48:23.874404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.874371 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerStarted","Data":"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633"} Nov 02 20:48:23.874897 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.874433 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerStarted","Data":"86a1bf6d110d570094cb97af11c6de2ba8c2d52b68f96a157d9106164def3f3b"} Nov 02 20:48:23.910582 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.908347 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:23.985235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:23.985173 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podStartSLOduration=1.9851576770000001 podStartE2EDuration="1.985157677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:23.984284656 +0000 UTC m=+3587.590829153" watchObservedRunningTime="2026-11-02 20:48:23.985157677 +0000 UTC m=+3587.591702174" Nov 02 20:48:24.913497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:24.913470 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:25.889011 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:25.888977 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:25.895758 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:25.890462 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:48:28.907423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:28.907388 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:48:28.911967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:28.911950 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:28.963180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:28.963147 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:48:29.006724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.006691 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a7c5d1-serving-cert\"" Nov 02 20:48:29.013485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.013458 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.013601 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.013514 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqm7j\" (UniqueName: \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.013642 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.013593 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.013725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.013712 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.115130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.115100 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.115283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.115218 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.115283 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:48:29.115243 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-serving-cert: secret "maas-api-e2e-extmod-a-a7c5d1-serving-cert" not found Nov 02 20:48:29.115283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.115262 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqm7j\" (UniqueName: \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.115283 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.115281 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.115497 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:48:29.115300 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls podName:e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c nodeName:}" failed. No retries permitted until 2026-11-02 20:48:29.615282536 +0000 UTC m=+3593.221827012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls") pod "maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" (UID: "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c") : secret "maas-api-e2e-extmod-a-a7c5d1-serving-cert" not found Nov 02 20:48:29.115926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.115906 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.117540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.117523 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.171571 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.171467 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqm7j\" (UniqueName: \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.624430 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.624396 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.626774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.626746 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:29.826334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:29.826300 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:30.200157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:30.200129 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:48:30.200878 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:48:30.200852 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aa2dbc_c5b8_4d55_b59d_c0b944eef94c.slice/crio-592302b656a5ec1f7645eb627c7f0bfc11e58cf9a36290321557b7604331b966 WatchSource:0}: Error finding container 592302b656a5ec1f7645eb627c7f0bfc11e58cf9a36290321557b7604331b966: Status 404 returned error can't find the container with id 592302b656a5ec1f7645eb627c7f0bfc11e58cf9a36290321557b7604331b966 Nov 02 20:48:30.924367 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:30.924328 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" event={"ID":"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c","Type":"ContainerStarted","Data":"592302b656a5ec1f7645eb627c7f0bfc11e58cf9a36290321557b7604331b966"} Nov 02 20:48:31.931682 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:31.931646 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" event={"ID":"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c","Type":"ContainerStarted","Data":"3e69847ff5383dba9b7eaf6fe496d7f3d625c8b779089c115ea75cafb8c6632a"} Nov 02 20:48:31.932143 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:31.931712 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:32.045046 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:32.044997 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" podStartSLOduration=3.200119731 podStartE2EDuration="4.044982222s" totalImagesPullingTime="844.862491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:32.042498485 +0000 UTC m=+3595.649042984" watchObservedRunningTime="2026-11-02 20:48:32.044982222 +0000 UTC m=+3595.651526721" Nov 02 20:48:37.943552 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:48:37.943524 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:48:51.005228 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:48:51.005137 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:49:15.224193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.224163 5136 generic.go:356] "Generic (PLEG): container finished" podID="f6aec69e-5151-4573-b96e-a20661538fe2" containerID="359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5" exitCode=0 Nov 02 20:49:15.224777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.224248 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerDied","Data":"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5"} Nov 02 20:49:15.224777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.224773 5136 scope.go:122] "RemoveContainer" containerID="359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5" Nov 02 20:49:15.226428 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.226398 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:49:15.226508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.226468 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:49:15.226508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:15.226501 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:49:16.232455 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:16.232417 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerStarted","Data":"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0"} Nov 02 20:49:16.442657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:16.442627 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:49:16.443464 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:16.443439 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 20:49:16.443584 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:16.443488 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 20:49:17.443341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:17.443309 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 20:49:17.443759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:17.443372 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 20:49:18.443058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:18.443016 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 20:49:18.443233 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:18.443087 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 20:49:19.443075 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:19.443039 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 20:49:19.443503 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:19.443104 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 20:49:20.256205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:20.256171 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:49:20.443395 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:20.443360 5136 patch_prober.go:29] interesting pod/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 20:49:20.443824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:20.443411 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 20:49:21.027296 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:49:21.027262 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:49:21.447155 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:21.447120 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:49:21.448132 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:21.448112 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:49:51.003980 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:49:51.003944 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:49:57.163167 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.163137 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:49:57.166415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.166395 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.263700 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.263665 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-020389-serving-cert\"" Nov 02 20:49:57.272315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.269796 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:49:57.287595 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.287564 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.287841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.287821 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.287918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.287869 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.287993 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.287977 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dx7\" (UniqueName: \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.389114 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.389078 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.389312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.389126 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.389312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.389159 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dx7\" (UniqueName: \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.389312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.389198 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.389312 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:49:57.389242 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-serving-cert: secret "maas-api-e2e-extmod-b-020389-serving-cert" not found Nov 02 20:49:57.389555 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:49:57.389329 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls podName:e46746f6-2fde-43d7-ae1a-bd16219f7bc1 nodeName:}" failed. No retries permitted until 2026-11-02 20:49:57.889306729 +0000 UTC m=+3681.495851227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls") pod "maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" (UID: "e46746f6-2fde-43d7-ae1a-bd16219f7bc1") : secret "maas-api-e2e-extmod-b-020389-serving-cert" not found Nov 02 20:49:57.389822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.389798 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.391573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.391548 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.446526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.446390 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dx7\" (UniqueName: \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.895204 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.895113 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:57.897545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:57.897519 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:58.078929 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:58.078899 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:58.444277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:58.444251 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:49:58.445642 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:49:58.445609 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46746f6_2fde_43d7_ae1a_bd16219f7bc1.slice/crio-3b5c9cb42242f1ece0d492e3c32727b1c1db7c0788d9010543efa8ec7cb3a7d1 WatchSource:0}: Error finding container 3b5c9cb42242f1ece0d492e3c32727b1c1db7c0788d9010543efa8ec7cb3a7d1: Status 404 returned error can't find the container with id 3b5c9cb42242f1ece0d492e3c32727b1c1db7c0788d9010543efa8ec7cb3a7d1 Nov 02 20:49:58.496181 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:58.496148 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" event={"ID":"e46746f6-2fde-43d7-ae1a-bd16219f7bc1","Type":"ContainerStarted","Data":"3b5c9cb42242f1ece0d492e3c32727b1c1db7c0788d9010543efa8ec7cb3a7d1"} Nov 02 20:49:59.507808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:59.507773 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" event={"ID":"e46746f6-2fde-43d7-ae1a-bd16219f7bc1","Type":"ContainerStarted","Data":"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7"} Nov 02 20:49:59.508215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:59.507886 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:49:59.642052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:49:59.641958 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" podStartSLOduration=1.726115627 podStartE2EDuration="2.641944577s" totalImagesPullingTime="915.82895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:49:59.639850669 +0000 UTC m=+3683.246395180" watchObservedRunningTime="2026-11-02 20:49:59.641944577 +0000 UTC m=+3683.248489128" Nov 02 20:50:05.518976 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:05.518948 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:50:07.047174 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.047134 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:50:07.050721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.050699 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.100562 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.100529 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-27e421-serving-cert\"" Nov 02 20:50:07.153068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.153026 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:50:07.216327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.216297 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.216560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.216537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpljr\" (UniqueName: \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.216607 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.216587 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.216668 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.216656 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.318899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.318087 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpljr\" (UniqueName: \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.318899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.318151 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.318899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.318196 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.318899 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.318252 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.319744 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.319701 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.322446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.322373 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.325276 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.325250 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.373561 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.373533 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpljr\" (UniqueName: \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\") pod \"maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:07.663292 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:07.663200 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:08.029610 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:08.029580 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:50:08.030530 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:50:08.030493 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b8553f_2c24_4d64_9c23_b683d81d3dbf.slice/crio-ff9568871241759546d8e6139e5a19dc38303ee7beec1c6f0a81603078bad802 WatchSource:0}: Error finding container ff9568871241759546d8e6139e5a19dc38303ee7beec1c6f0a81603078bad802: Status 404 returned error can't find the container with id ff9568871241759546d8e6139e5a19dc38303ee7beec1c6f0a81603078bad802 Nov 02 20:50:08.563158 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:08.563118 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" event={"ID":"76b8553f-2c24-4d64-9c23-b683d81d3dbf","Type":"ContainerStarted","Data":"ff9568871241759546d8e6139e5a19dc38303ee7beec1c6f0a81603078bad802"} Nov 02 20:50:09.570617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:09.570575 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" event={"ID":"76b8553f-2c24-4d64-9c23-b683d81d3dbf","Type":"ContainerStarted","Data":"879e51b2ee3fc57bf3c6daa55a7e03b44db1c987ce7408fcee38424278a41bca"} Nov 02 20:50:09.571045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:09.570752 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:09.680678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:09.680617 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" podStartSLOduration=1.8926685540000001 podStartE2EDuration="2.68060313s" totalImagesPullingTime="787.934576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:09.679460797 +0000 UTC m=+3693.286005297" watchObservedRunningTime="2026-11-02 20:50:09.68060313 +0000 UTC m=+3693.287147628" Nov 02 20:50:15.610781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:15.609101 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:50:19.715606 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:19.715513 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:50:21.061948 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:50:21.061909 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:50:24.716227 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:24.716192 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" podUID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" containerName="payload-pre-processing" containerID="cri-o://fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2" gracePeriod=25 Nov 02 20:50:24.991127 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:24.991099 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:50:25.021519 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.021471 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\") pod \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " Nov 02 20:50:25.021725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.021704 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\" (UniqueName: \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\") pod \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\" (UID: \"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4\") " Nov 02 20:50:25.021968 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.021841 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume" pod "e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" (UID: "e29f43a7-26a4-4d7f-9c9b-7afcf68051c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:50:25.022262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.022240 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:25.024526 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.024495 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw" pod "e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" (UID: "e29f43a7-26a4-4d7f-9c9b-7afcf68051c4"). InnerVolumeSpecName "kube-api-access-qfjtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:25.123634 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.123600 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjtw\" (UniqueName: \"kubernetes.io/projected/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4-kube-api-access-qfjtw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:25.684403 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.684372 5136 generic.go:356] "Generic (PLEG): container finished" podID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" containerID="fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2" exitCode=0 Nov 02 20:50:25.684592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.684422 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" Nov 02 20:50:25.684592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.684430 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" event={"ID":"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4","Type":"ContainerDied","Data":"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2"} Nov 02 20:50:25.684592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.684471 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m" event={"ID":"e29f43a7-26a4-4d7f-9c9b-7afcf68051c4","Type":"ContainerDied","Data":"9bc6493e0d52aea9f4076dd3755a50db442a2b837a8372e5ff0c42e77b5c3c55"} Nov 02 20:50:25.684592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.684496 5136 scope.go:122] "RemoveContainer" containerID="fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2" Nov 02 20:50:25.694740 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.694711 5136 scope.go:122] "RemoveContainer" containerID="fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2" Nov 02 20:50:25.695165 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:50:25.695143 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2\": container with ID starting with fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2 not found: ID does not exist" containerID="fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2" Nov 02 20:50:25.695277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.695174 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2"} err="failed to get container status \"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2\": rpc error: code = NotFound desc = could not find container \"fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2\": container with ID starting with fbea5119ff01e6639aa3a5b0156c5c243778213621f276da480255b5096649d2 not found: ID does not exist" Nov 02 20:50:25.841368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.841337 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:50:25.889285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:25.889254 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a7c5d1-5cf84df4c-vhk4m"] Nov 02 20:50:26.550423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:50:26.550388 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" path="/var/lib/kubelet/pods/e29f43a7-26a4-4d7f-9c9b-7afcf68051c4/volumes" Nov 02 20:50:51.004662 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:50:51.004625 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:51:01.942478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.942374 5136 generic.go:356] "Generic (PLEG): container finished" podID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerID="7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633" exitCode=0 Nov 02 20:51:01.942478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.942449 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerDied","Data":"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633"} Nov 02 20:51:01.943029 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.943009 5136 scope.go:122] "RemoveContainer" containerID="7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633" Nov 02 20:51:01.945325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.945292 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:51:01.945440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.945386 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:51:01.945506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:01.945439 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:51:02.950031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:02.949994 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerStarted","Data":"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85"} Nov 02 20:51:03.913842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:03.913799 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:51:03.914092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:03.914055 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:51:03.914187 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:03.914096 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 20:51:04.912117 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:04.912071 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:51:04.912724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:04.912683 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 20:51:05.911078 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:05.911035 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:51:05.911382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:05.911348 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 20:51:06.911061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:06.911018 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:51:06.912239 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:06.912195 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 20:51:07.911755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:07.910832 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:51:07.911755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:07.910895 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 20:51:08.913535 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:08.913504 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:51:08.914061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:08.913951 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:51:08.914969 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:08.914950 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:51:21.004796 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:51:21.004761 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:51:38.443128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:38.443091 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:51:43.444125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.444074 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" podUID="825b8950-821e-4778-8c18-840bdc5e11f8" containerName="payload-pre-processing" containerID="cri-o://c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25" gracePeriod=25 Nov 02 20:51:43.716050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.716023 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:51:43.852795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.852757 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\" (UniqueName: \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\") pod \"825b8950-821e-4778-8c18-840bdc5e11f8\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " Nov 02 20:51:43.852995 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.852869 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\") pod \"825b8950-821e-4778-8c18-840bdc5e11f8\" (UID: \"825b8950-821e-4778-8c18-840bdc5e11f8\") " Nov 02 20:51:43.853314 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.853286 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume" pod "825b8950-821e-4778-8c18-840bdc5e11f8" (UID: "825b8950-821e-4778-8c18-840bdc5e11f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:43.854876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.854852 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v" pod "825b8950-821e-4778-8c18-840bdc5e11f8" (UID: "825b8950-821e-4778-8c18-840bdc5e11f8"). InnerVolumeSpecName "kube-api-access-lks9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:43.954150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.954116 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lks9v\" (UniqueName: \"kubernetes.io/projected/825b8950-821e-4778-8c18-840bdc5e11f8-kube-api-access-lks9v\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:43.954150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:43.954150 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825b8950-821e-4778-8c18-840bdc5e11f8-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:44.216202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.216099 5136 generic.go:356] "Generic (PLEG): container finished" podID="825b8950-821e-4778-8c18-840bdc5e11f8" containerID="c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25" exitCode=0 Nov 02 20:51:44.216202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.216153 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" Nov 02 20:51:44.216202 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.216187 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" event={"ID":"825b8950-821e-4778-8c18-840bdc5e11f8","Type":"ContainerDied","Data":"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25"} Nov 02 20:51:44.216449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.216268 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc" event={"ID":"825b8950-821e-4778-8c18-840bdc5e11f8","Type":"ContainerDied","Data":"e2299b8685711d5af25f2923727e14ea7c668e0fe717b056cda86ef0ffb4fc82"} Nov 02 20:51:44.216449 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.216292 5136 scope.go:122] "RemoveContainer" containerID="c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25" Nov 02 20:51:44.226884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.226866 5136 scope.go:122] "RemoveContainer" containerID="c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25" Nov 02 20:51:44.227173 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:51:44.227150 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25\": container with ID starting with c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25 not found: ID does not exist" containerID="c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25" Nov 02 20:51:44.227236 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.227181 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25"} err="failed to get container status \"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25\": rpc error: code = NotFound desc = could not find container \"c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25\": container with ID starting with c86024a3ded00f4b7e2e07d46ee273db8bee748a05f5d2748702a04d058d7a25 not found: ID does not exist" Nov 02 20:51:44.373101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.373067 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:51:44.419861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.419832 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d5148-f8c7f8955-z48rc"] Nov 02 20:51:44.547776 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:44.547684 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825b8950-821e-4778-8c18-840bdc5e11f8" path="/var/lib/kubelet/pods/825b8950-821e-4778-8c18-840bdc5e11f8/volumes" Nov 02 20:51:51.004967 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:51:51.004894 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:51:53.101753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.099194 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:51:53.101753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.099493 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" podUID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" containerName="maas-api" containerID="cri-o://3e69847ff5383dba9b7eaf6fe496d7f3d625c8b779089c115ea75cafb8c6632a" gracePeriod=30 Nov 02 20:51:53.275144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.275114 5136 generic.go:356] "Generic (PLEG): container finished" podID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" containerID="3e69847ff5383dba9b7eaf6fe496d7f3d625c8b779089c115ea75cafb8c6632a" exitCode=0 Nov 02 20:51:53.275323 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.275193 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" event={"ID":"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c","Type":"ContainerDied","Data":"3e69847ff5383dba9b7eaf6fe496d7f3d625c8b779089c115ea75cafb8c6632a"} Nov 02 20:51:53.442869 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.442837 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:51:53.479753 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.479708 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\" (UniqueName: \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\") pod \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " Nov 02 20:51:53.479963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.479812 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\") pod \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " Nov 02 20:51:53.480039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.480003 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\") pod \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " Nov 02 20:51:53.480101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.480074 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") pod \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\" (UID: \"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c\") " Nov 02 20:51:53.480205 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.480164 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca" pod "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" (UID: "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:53.480665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.480639 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:53.482014 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.481993 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs" pod "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" (UID: "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:51:53.482110 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.482071 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j" pod "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" (UID: "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c"). InnerVolumeSpecName "kube-api-access-fqm7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:53.482163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.482128 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls" pod "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" (UID: "e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:51:53.585058 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.585018 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqm7j\" (UniqueName: \"kubernetes.io/projected/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-kube-api-access-fqm7j\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:53.585297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.585276 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:53.585425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:53.585401 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:54.283420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.283389 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" Nov 02 20:51:54.283420 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.283401 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj" event={"ID":"e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c","Type":"ContainerDied","Data":"592302b656a5ec1f7645eb627c7f0bfc11e58cf9a36290321557b7604331b966"} Nov 02 20:51:54.284061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.283454 5136 scope.go:122] "RemoveContainer" containerID="3e69847ff5383dba9b7eaf6fe496d7f3d625c8b779089c115ea75cafb8c6632a" Nov 02 20:51:54.439259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.439224 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:51:54.486843 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.486802 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a7c5d1-84f8fcd67-v7ctj"] Nov 02 20:51:54.547048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:54.546972 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" path="/var/lib/kubelet/pods/e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c/volumes" Nov 02 20:51:55.561098 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:55.561065 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:51:55.561903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:51:55.561870 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" containerID="cri-o://a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0" gracePeriod=30 Nov 02 20:52:00.833544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.833484 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:52:00.920043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920010 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920078 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920122 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920157 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920175 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920259 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920206 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920380 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920529 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920512 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920572 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:00.920754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920684 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:00.920885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920758 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:00.920885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920591 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\") pod \"f6aec69e-5151-4573-b96e-a20661538fe2\" (UID: \"f6aec69e-5151-4573-b96e-a20661538fe2\") " Nov 02 20:52:00.920885 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.920579 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:00.921075 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921052 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:00.921329 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921313 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:00.921418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921331 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6aec69e-5151-4573-b96e-a20661538fe2-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:00.921418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921340 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:00.921418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921349 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:00.921418 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.921359 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:00.922495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.922473 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:52:00.922741 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.922710 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:00.922949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.922928 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:00.923010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:00.922962 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6" pod "f6aec69e-5151-4573-b96e-a20661538fe2" (UID: "f6aec69e-5151-4573-b96e-a20661538fe2"). InnerVolumeSpecName "kube-api-access-l74v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:01.022491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.022453 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6aec69e-5151-4573-b96e-a20661538fe2-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:01.022491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.022482 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:01.022491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.022492 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6aec69e-5151-4573-b96e-a20661538fe2-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:01.022491 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.022503 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74v6\" (UniqueName: \"kubernetes.io/projected/f6aec69e-5151-4573-b96e-a20661538fe2-kube-api-access-l74v6\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:01.349920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.349888 5136 generic.go:356] "Generic (PLEG): container finished" podID="f6aec69e-5151-4573-b96e-a20661538fe2" containerID="a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0" exitCode=0 Nov 02 20:52:01.350097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.349945 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" Nov 02 20:52:01.350097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.349976 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerDied","Data":"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0"} Nov 02 20:52:01.350097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.350036 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz" event={"ID":"f6aec69e-5151-4573-b96e-a20661538fe2","Type":"ContainerDied","Data":"6138890b4273f24e7e860e640a40d0d3b8dac65aa68b39f48233a999071d1f5a"} Nov 02 20:52:01.350097 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.350062 5136 scope.go:122] "RemoveContainer" containerID="a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0" Nov 02 20:52:01.361474 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.361450 5136 scope.go:122] "RemoveContainer" containerID="359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5" Nov 02 20:52:01.371514 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.371490 5136 scope.go:122] "RemoveContainer" containerID="a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0" Nov 02 20:52:01.371808 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:52:01.371787 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0\": container with ID starting with a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0 not found: ID does not exist" containerID="a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0" Nov 02 20:52:01.371866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.371817 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0"} err="failed to get container status \"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0\": rpc error: code = NotFound desc = could not find container \"a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0\": container with ID starting with a2acff8aa84a3bacd280488f8e33ac2730513294717f68ea8019346bf8ff90f0 not found: ID does not exist" Nov 02 20:52:01.371866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.371836 5136 scope.go:122] "RemoveContainer" containerID="359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5" Nov 02 20:52:01.372108 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:52:01.372089 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5\": container with ID starting with 359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5 not found: ID does not exist" containerID="359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5" Nov 02 20:52:01.372177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.372121 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5"} err="failed to get container status \"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5\": rpc error: code = NotFound desc = could not find container \"359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5\": container with ID starting with 359e8c9ce0bb5a1006beae401047037d689aee1dd6bf5599960fdeebc63528a5 not found: ID does not exist" Nov 02 20:52:01.511564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.511530 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:52:01.518191 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:01.518159 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a7c5d1-openshift-default-784c55465c-kssgz"] Nov 02 20:52:02.547709 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:52:02.547673 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" path="/var/lib/kubelet/pods/f6aec69e-5151-4573-b96e-a20661538fe2/volumes" Nov 02 20:52:21.005786 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:52:21.005711 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:52:51.006011 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:52:51.005962 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:53:00.621285 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:00.621249 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:53:00.621988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:00.621461 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" podUID="431fe126-f76f-4149-8815-c86e19275ae9" containerName="maas-api" containerID="cri-o://3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f" gracePeriod=30 Nov 02 20:53:00.931834 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:00.931801 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:53:01.029970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.029931 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\") pod \"431fe126-f76f-4149-8815-c86e19275ae9\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " Nov 02 20:53:01.030497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.030460 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\") pod \"431fe126-f76f-4149-8815-c86e19275ae9\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " Nov 02 20:53:01.030706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.030514 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\" (UniqueName: \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\") pod \"431fe126-f76f-4149-8815-c86e19275ae9\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " Nov 02 20:53:01.030706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.030557 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\") pod \"431fe126-f76f-4149-8815-c86e19275ae9\" (UID: \"431fe126-f76f-4149-8815-c86e19275ae9\") " Nov 02 20:53:01.053757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.050987 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca" pod "431fe126-f76f-4149-8815-c86e19275ae9" (UID: "431fe126-f76f-4149-8815-c86e19275ae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:01.053757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.051061 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs" pod "431fe126-f76f-4149-8815-c86e19275ae9" (UID: "431fe126-f76f-4149-8815-c86e19275ae9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:53:01.072407 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.072368 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx" pod "431fe126-f76f-4149-8815-c86e19275ae9" (UID: "431fe126-f76f-4149-8815-c86e19275ae9"). InnerVolumeSpecName "kube-api-access-m6hzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:01.073800 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.073764 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls" pod "431fe126-f76f-4149-8815-c86e19275ae9" (UID: "431fe126-f76f-4149-8815-c86e19275ae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:53:01.132147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.132095 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431fe126-f76f-4149-8815-c86e19275ae9-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:01.132147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.132140 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hzx\" (UniqueName: \"kubernetes.io/projected/431fe126-f76f-4149-8815-c86e19275ae9-kube-api-access-m6hzx\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:01.132147 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.132154 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:01.132405 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.132163 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431fe126-f76f-4149-8815-c86e19275ae9-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:01.740257 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.740222 5136 generic.go:356] "Generic (PLEG): container finished" podID="431fe126-f76f-4149-8815-c86e19275ae9" containerID="3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f" exitCode=0 Nov 02 20:53:01.740792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.740269 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" Nov 02 20:53:01.740792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.740302 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" event={"ID":"431fe126-f76f-4149-8815-c86e19275ae9","Type":"ContainerDied","Data":"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f"} Nov 02 20:53:01.740792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.740360 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6" event={"ID":"431fe126-f76f-4149-8815-c86e19275ae9","Type":"ContainerDied","Data":"1e69c98d97cac3e2a6e6dd212cd5280a1990ca657659b83ba4d9e75b76e69134"} Nov 02 20:53:01.740792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.740391 5136 scope.go:122] "RemoveContainer" containerID="3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f" Nov 02 20:53:01.750948 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.750922 5136 scope.go:122] "RemoveContainer" containerID="3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f" Nov 02 20:53:01.751285 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:01.751257 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f\": container with ID starting with 3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f not found: ID does not exist" containerID="3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f" Nov 02 20:53:01.751351 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.751299 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f"} err="failed to get container status \"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f\": rpc error: code = NotFound desc = could not find container \"3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f\": container with ID starting with 3948ea7d8a6ee910de8b178d6b72fabc942b11c5b903dcc1aa1876752a9c825f not found: ID does not exist" Nov 02 20:53:01.896327 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.896289 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:53:01.899386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:01.899359 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d5148-67ccf5458-ddcm6"] Nov 02 20:53:02.548386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:02.548354 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431fe126-f76f-4149-8815-c86e19275ae9" path="/var/lib/kubelet/pods/431fe126-f76f-4149-8815-c86e19275ae9/volumes" Nov 02 20:53:03.527088 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:03.527052 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:53:03.527540 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:03.527329 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" containerID="cri-o://75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94" gracePeriod=30 Nov 02 20:53:08.789188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.789160 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:53:08.789997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.789977 5136 generic.go:356] "Generic (PLEG): container finished" podID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerID="75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94" exitCode=0 Nov 02 20:53:08.790096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.790051 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerDied","Data":"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94"} Nov 02 20:53:08.790142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.790106 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" event={"ID":"81f3fd04-2c44-41f8-9796-7f704a9b466b","Type":"ContainerDied","Data":"318643183e80abfeb7345cb70513df5413b5c77a8798000088a6aecedda393bb"} Nov 02 20:53:08.790142 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.790127 5136 scope.go:122] "RemoveContainer" containerID="75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94" Nov 02 20:53:08.800171 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.800150 5136 scope.go:122] "RemoveContainer" containerID="6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb" Nov 02 20:53:08.809722 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.809698 5136 scope.go:122] "RemoveContainer" containerID="75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94" Nov 02 20:53:08.810087 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:08.810066 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94\": container with ID starting with 75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94 not found: ID does not exist" containerID="75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94" Nov 02 20:53:08.810149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.810098 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94"} err="failed to get container status \"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94\": rpc error: code = NotFound desc = could not find container \"75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94\": container with ID starting with 75fc00d8b23e6a6988de9c83ac17c0c38a3a26b6cef0ab47274b47955f4fde94 not found: ID does not exist" Nov 02 20:53:08.810149 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.810117 5136 scope.go:122] "RemoveContainer" containerID="6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb" Nov 02 20:53:08.810353 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:08.810336 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb\": container with ID starting with 6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb not found: ID does not exist" containerID="6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb" Nov 02 20:53:08.810400 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.810361 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb"} err="failed to get container status \"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb\": rpc error: code = NotFound desc = could not find container \"6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb\": container with ID starting with 6d1d7982a005169ff93620a873639e9b1402691ee22b016d3e81d673cac255fb not found: ID does not exist" Nov 02 20:53:08.870579 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870543 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870650 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870680 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870772 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870814 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870885 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870927 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.870990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870965 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.871209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.870991 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:08.871209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871069 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\") pod \"81f3fd04-2c44-41f8-9796-7f704a9b466b\" (UID: \"81f3fd04-2c44-41f8-9796-7f704a9b466b\") " Nov 02 20:53:08.871330 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871208 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:08.871388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871340 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:08.871560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871513 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:08.871639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871557 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:08.871809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871766 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.871809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871793 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.871959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871809 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.871959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871823 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.871959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.871838 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f3fd04-2c44-41f8-9796-7f704a9b466b-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.873310 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.873287 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:08.873423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.873319 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "kube-api-access-xbccm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:08.873423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.873391 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:08.873590 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.873564 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo" pod "81f3fd04-2c44-41f8-9796-7f704a9b466b" (UID: "81f3fd04-2c44-41f8-9796-7f704a9b466b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:53:08.972801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.972764 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.972801 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.972796 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbccm\" (UniqueName: \"kubernetes.io/projected/81f3fd04-2c44-41f8-9796-7f704a9b466b-kube-api-access-xbccm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.973020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.972809 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:08.973020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:08.972823 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f3fd04-2c44-41f8-9796-7f704a9b466b-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:09.798317 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:09.798288 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc" Nov 02 20:53:10.002241 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:10.002205 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:53:10.049472 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:10.049389 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d5148-openshift-default-7bc5bfbd89-f75fc"] Nov 02 20:53:10.547726 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:10.547689 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" path="/var/lib/kubelet/pods/81f3fd04-2c44-41f8-9796-7f704a9b466b/volumes" Nov 02 20:53:17.604923 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.604815 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:53:17.605435 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.605076 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" podUID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" containerName="maas-api" containerID="cri-o://879e51b2ee3fc57bf3c6daa55a7e03b44db1c987ce7408fcee38424278a41bca" gracePeriod=30 Nov 02 20:53:17.865586 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.865549 5136 generic.go:356] "Generic (PLEG): container finished" podID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" containerID="879e51b2ee3fc57bf3c6daa55a7e03b44db1c987ce7408fcee38424278a41bca" exitCode=0 Nov 02 20:53:17.865762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.865619 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" event={"ID":"76b8553f-2c24-4d64-9c23-b683d81d3dbf","Type":"ContainerDied","Data":"879e51b2ee3fc57bf3c6daa55a7e03b44db1c987ce7408fcee38424278a41bca"} Nov 02 20:53:17.891713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.891684 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:53:17.993393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.993351 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\") pod \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " Nov 02 20:53:17.993611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.993445 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\" (UniqueName: \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\") pod \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " Nov 02 20:53:17.993611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.993484 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\") pod \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " Nov 02 20:53:17.993611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.993557 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\") pod \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\" (UID: \"76b8553f-2c24-4d64-9c23-b683d81d3dbf\") " Nov 02 20:53:17.994000 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.993969 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca" pod "76b8553f-2c24-4d64-9c23-b683d81d3dbf" (UID: "76b8553f-2c24-4d64-9c23-b683d81d3dbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:17.994484 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.994459 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b8553f-2c24-4d64-9c23-b683d81d3dbf-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:17.995973 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.995942 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs" pod "76b8553f-2c24-4d64-9c23-b683d81d3dbf" (UID: "76b8553f-2c24-4d64-9c23-b683d81d3dbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:53:17.996106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.995980 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls" pod "76b8553f-2c24-4d64-9c23-b683d81d3dbf" (UID: "76b8553f-2c24-4d64-9c23-b683d81d3dbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:53:17.996456 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:17.996427 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr" pod "76b8553f-2c24-4d64-9c23-b683d81d3dbf" (UID: "76b8553f-2c24-4d64-9c23-b683d81d3dbf"). InnerVolumeSpecName "kube-api-access-xpljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:18.095713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.095677 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:18.095713 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.095707 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpljr\" (UniqueName: \"kubernetes.io/projected/76b8553f-2c24-4d64-9c23-b683d81d3dbf-kube-api-access-xpljr\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:18.095964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.095723 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b8553f-2c24-4d64-9c23-b683d81d3dbf-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:18.873528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.873439 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" Nov 02 20:53:18.873528 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.873480 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk" event={"ID":"76b8553f-2c24-4d64-9c23-b683d81d3dbf","Type":"ContainerDied","Data":"ff9568871241759546d8e6139e5a19dc38303ee7beec1c6f0a81603078bad802"} Nov 02 20:53:18.874064 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:18.873548 5136 scope.go:122] "RemoveContainer" containerID="879e51b2ee3fc57bf3c6daa55a7e03b44db1c987ce7408fcee38424278a41bca" Nov 02 20:53:19.032373 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:19.032339 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:53:19.081997 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:19.081957 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-27e421-96ccbc596-k6lgk"] Nov 02 20:53:20.547352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:20.547315 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" path="/var/lib/kubelet/pods/76b8553f-2c24-4d64-9c23-b683d81d3dbf/volumes" Nov 02 20:53:21.005835 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:21.005792 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:53:23.434549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:23.434513 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:53:23.434990 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:23.434716 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" containerID="cri-o://5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85" gracePeriod=30 Nov 02 20:53:23.916198 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:23.916163 5136 patch_prober.go:29] interesting pod/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:53:23.916386 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:23.916221 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:53:28.719084 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.719054 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:53:28.831815 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.831770 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.831851 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.831876 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832024 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.831912 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832091 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832216 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832194 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832234 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832250 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:28.832335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832275 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832306 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\") pod \"e5ec6eae-4d68-4630-9b47-3d5826c94192\" (UID: \"e5ec6eae-4d68-4630-9b47-3d5826c94192\") " Nov 02 20:53:28.832560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832375 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:28.832623 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832581 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:28.832724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832698 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:28.832818 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.832800 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:28.833039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.833011 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.833039 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.833034 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.833152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.833049 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec6eae-4d68-4630-9b47-3d5826c94192-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.833152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.833065 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.833152 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.833078 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.834295 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.834260 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:53:28.834416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.834341 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "kube-api-access-6nqtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:28.834521 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.834504 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:28.834658 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.834636 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy" pod "e5ec6eae-4d68-4630-9b47-3d5826c94192" (UID: "e5ec6eae-4d68-4630-9b47-3d5826c94192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:53:28.934364 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.934328 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.934364 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.934359 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nqtt\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-kube-api-access-6nqtt\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.934364 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.934370 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.934639 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.934380 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec6eae-4d68-4630-9b47-3d5826c94192-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:28.939061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.939024 5136 generic.go:356] "Generic (PLEG): container finished" podID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerID="5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85" exitCode=0 Nov 02 20:53:28.939061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.939061 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerDied","Data":"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85"} Nov 02 20:53:28.939281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.939074 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" Nov 02 20:53:28.939281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.939113 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp" event={"ID":"e5ec6eae-4d68-4630-9b47-3d5826c94192","Type":"ContainerDied","Data":"86a1bf6d110d570094cb97af11c6de2ba8c2d52b68f96a157d9106164def3f3b"} Nov 02 20:53:28.939281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.939142 5136 scope.go:122] "RemoveContainer" containerID="5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85" Nov 02 20:53:28.950680 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.950660 5136 scope.go:122] "RemoveContainer" containerID="7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633" Nov 02 20:53:28.959792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.959773 5136 scope.go:122] "RemoveContainer" containerID="5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85" Nov 02 20:53:28.960075 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:28.960053 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85\": container with ID starting with 5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85 not found: ID does not exist" containerID="5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85" Nov 02 20:53:28.960139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.960083 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85"} err="failed to get container status \"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85\": rpc error: code = NotFound desc = could not find container \"5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85\": container with ID starting with 5937bf27ebce7a81c044acc35f6112c3d22318cc2163f0792409a77f604a2f85 not found: ID does not exist" Nov 02 20:53:28.960139 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.960101 5136 scope.go:122] "RemoveContainer" containerID="7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633" Nov 02 20:53:28.960373 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:28.960354 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633\": container with ID starting with 7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633 not found: ID does not exist" containerID="7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633" Nov 02 20:53:28.960436 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:28.960384 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633"} err="failed to get container status \"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633\": rpc error: code = NotFound desc = could not find container \"7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633\": container with ID starting with 7e3ae2f7092aa573210ebd11e8ad2f87fe38ed8766e4e8d78a325d7d9d2ee633 not found: ID does not exist" Nov 02 20:53:29.057712 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.057680 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 20:53:29.058596 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058572 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.058665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058605 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.058665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058632 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.058665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058640 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.058665 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058659 5136 cpu_manager.go:414] "containerMap: removing container" podUID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058666 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058697 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" containerName="payload-pre-processing" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058708 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" containerName="payload-pre-processing" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058722 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058745 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058768 5136 cpu_manager.go:414] "containerMap: removing container" podUID="825b8950-821e-4778-8c18-840bdc5e11f8" containerName="payload-pre-processing" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058778 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825b8950-821e-4778-8c18-840bdc5e11f8" containerName="payload-pre-processing" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058797 5136 cpu_manager.go:414] "containerMap: removing container" podUID="431fe126-f76f-4149-8815-c86e19275ae9" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058806 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431fe126-f76f-4149-8815-c86e19275ae9" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058818 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058827 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058842 5136 cpu_manager.go:414] "containerMap: removing container" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058848 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058854 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058859 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" containerName="maas-api" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058878 5136 cpu_manager.go:414] "containerMap: removing container" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:53:29.058882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058882 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058967 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058975 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058984 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6aec69e-5151-4573-b96e-a20661538fe2" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058990 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b8553f-2c24-4d64-9c23-b683d81d3dbf" containerName="maas-api" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.058997 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29f43a7-26a4-4d7f-9c9b-7afcf68051c4" containerName="payload-pre-processing" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.059004 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.059010 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="825b8950-821e-4778-8c18-840bdc5e11f8" containerName="payload-pre-processing" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.059016 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="431fe126-f76f-4149-8815-c86e19275ae9" containerName="maas-api" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.059022 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" containerName="istio-proxy" Nov 02 20:53:29.059481 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.059029 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4aa2dbc-c5b8-4d55-b59d-c0b944eef94c" containerName="maas-api" Nov 02 20:53:29.064030 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.064009 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.099925 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.099894 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:53:29.105284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.105252 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-27e421-openshift-default-6ffd5cd7d9-x5sdp"] Nov 02 20:53:29.112787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.112761 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 20:53:29.113480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.113459 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8a438af1b0c\"" Nov 02 20:53:29.113480 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.113475 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 20:53:29.113638 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.113483 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 20:53:29.136508 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.136473 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.136686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.136553 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.136831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.136808 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.136949 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.136932 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmzq\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.144759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.144687 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lkhrv\"" Nov 02 20:53:29.144937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.144726 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 20:53:29.144937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.144833 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c233618dc3e5\"" Nov 02 20:53:29.144937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.144886 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2da8cddd53dd\"" Nov 02 20:53:29.144937 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.144894 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 20:53:29.237680 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.237598 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.237854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.237701 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmzq\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.237854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.237769 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.237854 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.237826 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.238561 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.238533 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.239052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.239023 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.240785 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.240765 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.291809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.291780 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmzq\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\") pod \"payload-processing-external-model-7d68449d5-wszs7\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.378465 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.378431 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.748201 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.748167 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 20:53:29.749299 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:53:29.749267 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda364c385_934d_4d00_ac4c_4c22ff3c4656.slice/crio-1a8b13ac7d4055c92468c043e5e3b7035e46e81b63a16572cac54cb80bc5a29d WatchSource:0}: Error finding container 1a8b13ac7d4055c92468c043e5e3b7035e46e81b63a16572cac54cb80bc5a29d: Status 404 returned error can't find the container with id 1a8b13ac7d4055c92468c043e5e3b7035e46e81b63a16572cac54cb80bc5a29d Nov 02 20:53:29.947187 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.947130 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" event={"ID":"a364c385-934d-4d00-ac4c-4c22ff3c4656","Type":"ContainerStarted","Data":"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c"} Nov 02 20:53:29.947390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.947212 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:29.947390 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:29.947237 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" event={"ID":"a364c385-934d-4d00-ac4c-4c22ff3c4656","Type":"ContainerStarted","Data":"1a8b13ac7d4055c92468c043e5e3b7035e46e81b63a16572cac54cb80bc5a29d"} Nov 02 20:53:30.054977 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:30.054882 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" podStartSLOduration=1.054866981 podStartE2EDuration="1.054866981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:53:30.054126095 +0000 UTC m=+3893.660670593" watchObservedRunningTime="2026-11-02 20:53:30.054866981 +0000 UTC m=+3893.661411478" Nov 02 20:53:30.548964 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:30.548929 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ec6eae-4d68-4630-9b47-3d5826c94192" path="/var/lib/kubelet/pods/e5ec6eae-4d68-4630-9b47-3d5826c94192/volumes" Nov 02 20:53:40.958160 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:53:40.958129 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 20:53:51.005142 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:53:51.005110 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:54:21.004911 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:21.004869 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:54:22.130821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.130783 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 20:54:22.131547 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.131531 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f3fd04-2c44-41f8-9796-7f704a9b466b" containerName="istio-proxy" Nov 02 20:54:22.135346 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.135323 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.210344 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.210310 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f97409-openshift-default-dockercfg-s44tm\"" Nov 02 20:54:22.249176 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.249142 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 20:54:22.265795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.265760 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.265960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.265808 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.265960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.265935 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.265967 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266059 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.265985 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.266070 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjhs\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266153 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.266091 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266266 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.266218 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.266312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.266303 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.371751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371537 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.371751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371607 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.371751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371692 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.371751 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371719 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.372194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371762 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.372194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371799 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjhs\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.372194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371832 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.372194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371913 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.372194 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.371969 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.374631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.372488 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.374631 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.372774 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.382839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.382760 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.383162 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.383139 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.383574 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.383555 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.386530 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.386506 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.393577 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.393531 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.436424 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.436386 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.437040 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.437015 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjhs\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\") pod \"e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.450903 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.450855 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:22.873165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.873133 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 20:54:22.873857 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:22.873832 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8e0a95_34f4_4355_bfc5_258b1da17a5f.slice/crio-b5b0fd6ef760cb788d3162f2a2a8acb496780b657818df027b3c854b3f540ff5 WatchSource:0}: Error finding container b5b0fd6ef760cb788d3162f2a2a8acb496780b657818df027b3c854b3f540ff5: Status 404 returned error can't find the container with id b5b0fd6ef760cb788d3162f2a2a8acb496780b657818df027b3c854b3f540ff5 Nov 02 20:54:22.876868 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.876833 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:22.876972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.876923 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:22.877045 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:22.876991 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:23.279223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.279190 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:54:23.279653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.279478 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" podUID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" containerName="maas-api" containerID="cri-o://b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde" gracePeriod=30 Nov 02 20:54:23.362497 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.362458 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerStarted","Data":"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850"} Nov 02 20:54:23.362673 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.362518 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerStarted","Data":"b5b0fd6ef760cb788d3162f2a2a8acb496780b657818df027b3c854b3f540ff5"} Nov 02 20:54:23.451070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.451042 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:23.458866 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.458836 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:23.508260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.508209 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podStartSLOduration=1.5081960140000001 podStartE2EDuration="1.508196014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:23.507383058 +0000 UTC m=+3947.113927556" watchObservedRunningTime="2026-11-02 20:54:23.508196014 +0000 UTC m=+3947.114740506" Nov 02 20:54:23.573918 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.573892 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:54:23.693229 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693197 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") pod \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " Nov 02 20:54:23.693423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693293 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\") pod \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " Nov 02 20:54:23.693423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693330 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\" (UniqueName: \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\") pod \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " Nov 02 20:54:23.693423 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693397 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\") pod \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\" (UID: \"47fa2c93-dc2a-4cde-9d63-1b8e31388366\") " Nov 02 20:54:23.693814 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693786 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca" pod "47fa2c93-dc2a-4cde-9d63-1b8e31388366" (UID: "47fa2c93-dc2a-4cde-9d63-1b8e31388366"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:23.693960 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.693943 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fa2c93-dc2a-4cde-9d63-1b8e31388366-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:23.695556 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.695533 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5" pod "47fa2c93-dc2a-4cde-9d63-1b8e31388366" (UID: "47fa2c93-dc2a-4cde-9d63-1b8e31388366"). InnerVolumeSpecName "kube-api-access-9wzf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:23.695892 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.695872 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls" pod "47fa2c93-dc2a-4cde-9d63-1b8e31388366" (UID: "47fa2c93-dc2a-4cde-9d63-1b8e31388366"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:23.695962 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.695900 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs" pod "47fa2c93-dc2a-4cde-9d63-1b8e31388366" (UID: "47fa2c93-dc2a-4cde-9d63-1b8e31388366"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:23.795324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.795287 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:23.795324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.795318 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fa2c93-dc2a-4cde-9d63-1b8e31388366-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:23.795324 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:23.795329 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzf5\" (UniqueName: \"kubernetes.io/projected/47fa2c93-dc2a-4cde-9d63-1b8e31388366-kube-api-access-9wzf5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:24.392144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.392099 5136 generic.go:356] "Generic (PLEG): container finished" podID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" containerID="b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde" exitCode=0 Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.393362 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.393554 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" event={"ID":"47fa2c93-dc2a-4cde-9d63-1b8e31388366","Type":"ContainerDied","Data":"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde"} Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.393624 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5" event={"ID":"47fa2c93-dc2a-4cde-9d63-1b8e31388366","Type":"ContainerDied","Data":"7ac413bf343ebb8ec48f47a4d22f8bb7ab8f84e9a06cb5715bafa9d7ec3fde84"} Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.393769 5136 scope.go:122] "RemoveContainer" containerID="b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde" Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.394582 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:24.394963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.394909 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:54:24.407391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.407367 5136 scope.go:122] "RemoveContainer" containerID="b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde" Nov 02 20:54:24.407804 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:24.407764 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde\": container with ID starting with b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde not found: ID does not exist" containerID="b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde" Nov 02 20:54:24.407908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.407811 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde"} err="failed to get container status \"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde\": rpc error: code = NotFound desc = could not find container \"b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde\": container with ID starting with b09bb569f0265ac68e79ded419e111417e898afcfd853e608a6fe63e00923cde not found: ID does not exist" Nov 02 20:54:24.661165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.661084 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:54:24.669690 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:24.669657 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0a8f62-869dd4765c-lltd5"] Nov 02 20:54:25.713975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:25.713940 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:54:25.714504 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:25.714163 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" containerID="cri-o://9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915" gracePeriod=30 Nov 02 20:54:26.547846 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:26.547809 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" path="/var/lib/kubelet/pods/47fa2c93-dc2a-4cde-9d63-1b8e31388366/volumes" Nov 02 20:54:27.053297 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.053265 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 20:54:27.053882 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.053867 5136 cpu_manager.go:414] "containerMap: removing container" podUID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" containerName="maas-api" Nov 02 20:54:27.053943 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.053882 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" containerName="maas-api" Nov 02 20:54:27.054036 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.054024 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fa2c93-dc2a-4cde-9d63-1b8e31388366" containerName="maas-api" Nov 02 20:54:27.058802 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.058777 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.156011 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.155971 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwwd\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.156280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.156250 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.156280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.156281 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.156425 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.156384 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.159985 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.159952 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 20:54:27.258106 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.258069 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwwd\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.258307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.258242 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.258307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.258264 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.258307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.258300 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.258967 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.258944 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.259071 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.259048 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.260819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.260793 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.312762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.312675 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwwd\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\") pod \"payload-processing-external-model-5fcf8d6949-lqfjb\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.373260 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.373220 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:27.751660 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.751630 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 20:54:27.753209 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:27.753180 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4429f2_5ab9_4205_9ff6_5828958c1c54.slice/crio-dde3c195d6eccd34bf0d143f474d1ad95ad2da5f0184db9f8dbca7ae44910d38 WatchSource:0}: Error finding container dde3c195d6eccd34bf0d143f474d1ad95ad2da5f0184db9f8dbca7ae44910d38: Status 404 returned error can't find the container with id dde3c195d6eccd34bf0d143f474d1ad95ad2da5f0184db9f8dbca7ae44910d38 Nov 02 20:54:27.876166 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.876130 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 20:54:27.880382 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.880362 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.971699 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.971656 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.971921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.971718 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.971981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.971939 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972020 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972144 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972127 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972176 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972288 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972268 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8972\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972347 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972402 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972382 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:27.972994 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:27.972972 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 20:54:28.073908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.073813 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.073908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.073861 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.073908 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.073894 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.073933 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074004 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8972\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074078 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074112 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074280 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074312 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074319 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074389 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074619 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.074919 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.074704 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.075100 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.075080 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.076188 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.076170 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.076453 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.076435 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.129196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.129161 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.129414 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.129250 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8972\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-4878k\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.195844 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.195804 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:28.425445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.425355 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" event={"ID":"4e4429f2-5ab9-4205-9ff6-5828958c1c54","Type":"ContainerStarted","Data":"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35"} Nov 02 20:54:28.425445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.425414 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" event={"ID":"4e4429f2-5ab9-4205-9ff6-5828958c1c54","Type":"ContainerStarted","Data":"dde3c195d6eccd34bf0d143f474d1ad95ad2da5f0184db9f8dbca7ae44910d38"} Nov 02 20:54:28.425685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.425574 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 20:54:28.536154 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.536093 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podStartSLOduration=1.536077545 podStartE2EDuration="1.536077545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:28.533722607 +0000 UTC m=+3952.140267105" watchObservedRunningTime="2026-11-02 20:54:28.536077545 +0000 UTC m=+3952.142622043" Nov 02 20:54:28.660417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.660387 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 20:54:28.661574 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:28.661539 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765c1df4_1f03_4847_b5a6_f4a075c40ee8.slice/crio-f99ea114b87abe9936eb1dbe19606581798056f51f1cc07cd618ef2904a9d635 WatchSource:0}: Error finding container f99ea114b87abe9936eb1dbe19606581798056f51f1cc07cd618ef2904a9d635: Status 404 returned error can't find the container with id f99ea114b87abe9936eb1dbe19606581798056f51f1cc07cd618ef2904a9d635 Nov 02 20:54:28.664809 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.664773 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:28.664927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.664860 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:28.664998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:28.664930 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:29.434163 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:29.434131 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" event={"ID":"765c1df4-1f03-4847-b5a6-f4a075c40ee8","Type":"ContainerStarted","Data":"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f"} Nov 02 20:54:29.434646 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:29.434184 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" event={"ID":"765c1df4-1f03-4847-b5a6-f4a075c40ee8","Type":"ContainerStarted","Data":"f99ea114b87abe9936eb1dbe19606581798056f51f1cc07cd618ef2904a9d635"} Nov 02 20:54:29.583406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:29.583344 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podStartSLOduration=2.583322307 podStartE2EDuration="2.583322307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:29.581964079 +0000 UTC m=+3953.188508586" watchObservedRunningTime="2026-11-02 20:54:29.583322307 +0000 UTC m=+3953.189866810" Nov 02 20:54:30.209755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:30.200660 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:30.209755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:30.200878 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:30.209755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:30.200924 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:31.057129 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.057100 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:54:31.144231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144197 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144246 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144267 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144314 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144378 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144429 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144401 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144470 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144492 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144573 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144641 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144674 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:31.144720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144702 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\") pod \"38bcf0f1-b043-4239-b2de-8766ebbe7072\" (UID: \"38bcf0f1-b043-4239-b2de-8766ebbe7072\") " Nov 02 20:54:31.145052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.144912 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:31.145301 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.145280 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.145392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.145310 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.145392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.145325 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.145392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.145340 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.145392 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.145281 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:31.146741 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.146670 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:31.146860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.146769 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:54:31.147109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.147089 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:31.147180 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.147095 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p" pod "38bcf0f1-b043-4239-b2de-8766ebbe7072" (UID: "38bcf0f1-b043-4239-b2de-8766ebbe7072"). InnerVolumeSpecName "kube-api-access-9q98p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:31.209440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.209024 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:31.209440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.209078 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:31.247026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.246987 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bcf0f1-b043-4239-b2de-8766ebbe7072-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.247026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.247022 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.247026 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.247032 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.247289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.247041 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q98p\" (UniqueName: \"kubernetes.io/projected/38bcf0f1-b043-4239-b2de-8766ebbe7072-kube-api-access-9q98p\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.247289 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.247054 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bcf0f1-b043-4239-b2de-8766ebbe7072-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:31.449506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.449468 5136 generic.go:356] "Generic (PLEG): container finished" podID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerID="9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915" exitCode=0 Nov 02 20:54:31.449970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.449953 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" Nov 02 20:54:31.450391 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.450348 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerDied","Data":"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915"} Nov 02 20:54:31.450467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.450400 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449" event={"ID":"38bcf0f1-b043-4239-b2de-8766ebbe7072","Type":"ContainerDied","Data":"22a4b9b1d84b2d8959e2c0e7a7db52a89aa55da60d6e9236ae5f45186114cf9d"} Nov 02 20:54:31.450467 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.450424 5136 scope.go:122] "RemoveContainer" containerID="9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915" Nov 02 20:54:31.461721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.461699 5136 scope.go:122] "RemoveContainer" containerID="bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128" Nov 02 20:54:31.471368 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.471347 5136 scope.go:122] "RemoveContainer" containerID="9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915" Nov 02 20:54:31.471636 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:31.471619 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915\": container with ID starting with 9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915 not found: ID does not exist" containerID="9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915" Nov 02 20:54:31.471688 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.471647 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915"} err="failed to get container status \"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915\": rpc error: code = NotFound desc = could not find container \"9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915\": container with ID starting with 9bae404c1fba96ce84f17192a21e1cbe3ee4576dcd358bc9c1d27d0c0f0cd915 not found: ID does not exist" Nov 02 20:54:31.471688 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.471670 5136 scope.go:122] "RemoveContainer" containerID="bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128" Nov 02 20:54:31.471958 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:31.471937 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128\": container with ID starting with bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128 not found: ID does not exist" containerID="bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128" Nov 02 20:54:31.472006 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.471962 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128"} err="failed to get container status \"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128\": rpc error: code = NotFound desc = could not find container \"bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128\": container with ID starting with bc6d19c772ee87f2aae3a9586ed619693acfdbbfcbe34fe35826686a7b339128 not found: ID does not exist" Nov 02 20:54:31.618416 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.618349 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:54:31.620813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:31.620791 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0a8f62-openshift-default-7b594fb789-5z449"] Nov 02 20:54:32.212787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.212235 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:32.212787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.212304 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:32.528445 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.528353 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 20:54:32.529123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529095 5136 cpu_manager.go:414] "containerMap: removing container" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.529123 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529118 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.529312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529166 5136 cpu_manager.go:414] "containerMap: removing container" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.529312 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529174 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.529426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529354 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.529426 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.529374 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" containerName="istio-proxy" Nov 02 20:54:32.532988 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.532961 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.548380 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.548354 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bcf0f1-b043-4239-b2de-8766ebbe7072" path="/var/lib/kubelet/pods/38bcf0f1-b043-4239-b2de-8766ebbe7072/volumes" Nov 02 20:54:32.573215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.573184 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz9w\" (UniqueName: \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.573345 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.573324 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.582909 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.582882 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f97409\"" Nov 02 20:54:32.583043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.582984 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f97409-dockercfg-zkx29\"" Nov 02 20:54:32.584895 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.584873 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 20:54:32.677782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.675900 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz9w\" (UniqueName: \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.677782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.676080 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.677782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.677163 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.698927 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.698884 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 20:54:32.710250 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.710223 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.734803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.733773 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz9w\" (UniqueName: \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\") pod \"payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.762995 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.762967 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 20:54:32.778060 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.778027 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.778245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.778220 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f9j\" (UniqueName: \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.847975 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.847886 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:32.879983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.879949 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.880177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.880030 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f9j\" (UniqueName: \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.880714 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.880688 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:32.948821 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:32.943204 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f9j\" (UniqueName: \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\") pod \"payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:33.038697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.038665 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:33.216867 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.216827 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:33.217359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.216888 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:33.257719 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:33.257259 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772c9378_13ae_44ec_93a1_24c271e49713.slice/crio-a432f493a7801fd05950fcc8c085907eb90513fa92fa123f3b7c862a73b028be WatchSource:0}: Error finding container a432f493a7801fd05950fcc8c085907eb90513fa92fa123f3b7c862a73b028be: Status 404 returned error can't find the container with id a432f493a7801fd05950fcc8c085907eb90513fa92fa123f3b7c862a73b028be Nov 02 20:54:33.258070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.257967 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 20:54:33.439678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.439651 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 20:54:33.441060 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:33.441026 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b94f48_d953_458f_be3b_2505a67c7538.slice/crio-e95d876204997eac1c09dccf439352eb06f9947401f94b0428b3d0ad9d1799ed WatchSource:0}: Error finding container e95d876204997eac1c09dccf439352eb06f9947401f94b0428b3d0ad9d1799ed: Status 404 returned error can't find the container with id e95d876204997eac1c09dccf439352eb06f9947401f94b0428b3d0ad9d1799ed Nov 02 20:54:33.479693 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.479632 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" event={"ID":"85b94f48-d953-458f-be3b-2505a67c7538","Type":"ContainerStarted","Data":"e95d876204997eac1c09dccf439352eb06f9947401f94b0428b3d0ad9d1799ed"} Nov 02 20:54:33.481243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.481222 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" event={"ID":"772c9378-13ae-44ec-93a1-24c271e49713","Type":"ContainerStarted","Data":"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4"} Nov 02 20:54:33.481388 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.481374 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" event={"ID":"772c9378-13ae-44ec-93a1-24c271e49713","Type":"ContainerStarted","Data":"a432f493a7801fd05950fcc8c085907eb90513fa92fa123f3b7c862a73b028be"} Nov 02 20:54:33.481470 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:33.481458 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:34.204756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.201187 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:34.204756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.201248 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:34.512256 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.512164 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" event={"ID":"85b94f48-d953-458f-be3b-2505a67c7538","Type":"ContainerStarted","Data":"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d"} Nov 02 20:54:34.512864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.512838 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:34.627617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.624941 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" podStartSLOduration=2.624920878 podStartE2EDuration="2.624920878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:34.622122903 +0000 UTC m=+3958.228667401" watchObservedRunningTime="2026-11-02 20:54:34.624920878 +0000 UTC m=+3958.231465377" Nov 02 20:54:34.633361 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:34.633302 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" podStartSLOduration=2.633285275 podStartE2EDuration="2.633285275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:33.607088618 +0000 UTC m=+3957.213633118" watchObservedRunningTime="2026-11-02 20:54:34.633285275 +0000 UTC m=+3958.239829773" Nov 02 20:54:35.197214 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:35.197170 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:35.197458 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:35.197235 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:36.198813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:36.198767 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:54:36.199186 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:36.198833 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:54:37.401489 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:37.401460 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:37.402245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:37.402211 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:37.403107 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:37.403090 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 20:54:39.411676 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.411645 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 20:54:39.417291 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.417269 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.439222 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.439178 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:54:39.467534 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.467503 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 20:54:39.520089 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.520054 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f3109b\"" Nov 02 20:54:39.520459 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.520443 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f3109b-dockercfg-jbqp9\"" Nov 02 20:54:39.543338 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.543298 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.543822 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.543799 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8v7\" (UniqueName: \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.572774 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.572724 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 20:54:39.578213 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.578185 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.629828 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.629790 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 20:54:39.645012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.644981 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8v7\" (UniqueName: \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.645219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.645017 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.645219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.645075 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6ck\" (UniqueName: \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.645219 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.645119 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.646068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.646046 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.699270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.699237 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8v7\" (UniqueName: \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\") pod \"payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.732161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.732124 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:39.745958 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.745929 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.746093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.746001 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6ck\" (UniqueName: \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.746667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.746642 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.810781 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.810674 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6ck\" (UniqueName: \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\") pod \"payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:39.893599 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:39.893562 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:40.100280 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.100252 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 20:54:40.100893 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:40.100870 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68c87a_2391_4e67_a202_7804a6c1f380.slice/crio-d0570839f9fa0cf691acd5f7e6286c3261a0442f64881a682689fab23340ed7d WatchSource:0}: Error finding container d0570839f9fa0cf691acd5f7e6286c3261a0442f64881a682689fab23340ed7d: Status 404 returned error can't find the container with id d0570839f9fa0cf691acd5f7e6286c3261a0442f64881a682689fab23340ed7d Nov 02 20:54:40.269093 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:54:40.269057 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8a4baf_a24b_4cb5_a1d4_55c08df2a432.slice/crio-5b8e0bc4a2052531b9e58334d0a76704c9cb26c3000049e6f699b982b5b996f4 WatchSource:0}: Error finding container 5b8e0bc4a2052531b9e58334d0a76704c9cb26c3000049e6f699b982b5b996f4: Status 404 returned error can't find the container with id 5b8e0bc4a2052531b9e58334d0a76704c9cb26c3000049e6f699b982b5b996f4 Nov 02 20:54:40.269209 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.269194 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 20:54:40.561309 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.561225 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" event={"ID":"ef68c87a-2391-4e67-a202-7804a6c1f380","Type":"ContainerStarted","Data":"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723"} Nov 02 20:54:40.561309 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.561277 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" event={"ID":"ef68c87a-2391-4e67-a202-7804a6c1f380","Type":"ContainerStarted","Data":"d0570839f9fa0cf691acd5f7e6286c3261a0442f64881a682689fab23340ed7d"} Nov 02 20:54:40.561865 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.561317 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:40.562983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.562956 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" event={"ID":"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432","Type":"ContainerStarted","Data":"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50"} Nov 02 20:54:40.563096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.563020 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" event={"ID":"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432","Type":"ContainerStarted","Data":"5b8e0bc4a2052531b9e58334d0a76704c9cb26c3000049e6f699b982b5b996f4"} Nov 02 20:54:40.563096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.563044 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:40.671130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.671072 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" podStartSLOduration=1.67105825 podStartE2EDuration="1.67105825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:40.669984893 +0000 UTC m=+3964.276529393" watchObservedRunningTime="2026-11-02 20:54:40.67105825 +0000 UTC m=+3964.277602747" Nov 02 20:54:40.778164 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:40.778098 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" podStartSLOduration=1.778076725 podStartE2EDuration="1.778076725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:40.776522673 +0000 UTC m=+3964.383067172" watchObservedRunningTime="2026-11-02 20:54:40.778076725 +0000 UTC m=+3964.384621223" Nov 02 20:54:42.589787 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:42.580886 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:42.589787 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:42.580918 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:42.589787 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:42.580932 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:42.589787 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:42.580998 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:54:43.08098022 +0000 UTC m=+3966.687524716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:42.990647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:42.990601 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 20:54:42.996096 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:42.996074 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.083681 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.083642 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 20:54:43.085168 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085137 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085318 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085199 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085357 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085446 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085410 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68rk\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085557 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085628 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085613 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085774 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085870 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085846 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.085916 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.085891 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.086103 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:43.086077 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:43.086209 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:43.086110 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:43.086209 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:43.086126 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:43.086209 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:43.086177 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:54:44.086162075 +0000 UTC m=+3967.692706570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:43.187618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187576 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.187618 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187623 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.187938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187687 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.187938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187723 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68rk\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.187938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187784 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.187938 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187822 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187961 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.187991 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188024 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188084 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188183 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188097 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188478 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188230 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188544 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188492 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.188862 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.188838 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.190406 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.190384 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.190522 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.190506 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.241947 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.241864 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.242124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.241953 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68rk\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mtzhz\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.315720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.315670 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:43.753215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.752401 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:43.753215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.752501 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:43.753215 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.752545 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:54:43.755861 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:43.755696 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 20:54:44.123813 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:44.123017 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:44.123813 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:44.123067 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:44.123813 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:44.123081 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:44.123813 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:44.123180 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:54:46.123159802 +0000 UTC m=+3969.729704297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:44.602756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:44.600086 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" event={"ID":"02248d36-3a0a-40d1-8d1f-95c6198e4247","Type":"ContainerStarted","Data":"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520"} Nov 02 20:54:44.602756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:44.600187 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" event={"ID":"02248d36-3a0a-40d1-8d1f-95c6198e4247","Type":"ContainerStarted","Data":"8541208bcd6bb4e977bbffc8276ee801fa29fb8ef69e2f800e6ecea26656ecc0"} Nov 02 20:54:44.726754 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:44.726543 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podStartSLOduration=2.72652452 podStartE2EDuration="2.72652452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:44.726072355 +0000 UTC m=+3968.332616855" watchObservedRunningTime="2026-11-02 20:54:44.72652452 +0000 UTC m=+3968.333069013" Nov 02 20:54:45.317755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:45.316882 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:45.318249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:45.317993 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:45.318249 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:45.318083 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:45.522315 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:45.522282 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 20:54:45.522479 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:45.522348 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 20:54:46.173443 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:46.173408 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:46.173443 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:46.173443 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:46.173443 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:46.173452 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:46.173686 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:46.173515 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:54:50.17349912 +0000 UTC m=+3973.780043620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:46.316816 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:46.316781 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:46.316999 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:46.316840 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:47.316376 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:47.316342 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:47.316795 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:47.316408 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:48.316444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:48.316409 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:48.316850 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:48.316461 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:49.316549 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:49.316499 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:49.317087 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:49.316553 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:49.436404 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:49.436372 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:54:50.252012 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:50.251971 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:50.252012 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:50.252006 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:50.252012 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:50.252015 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:50.252291 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:50.252068 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:54:58.252054287 +0000 UTC m=+3981.858598764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:50.316385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:50.316349 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:50.316545 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:50.316402 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:51.006064 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:51.006017 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:54:51.317161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:51.317077 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:54:51.317161 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:51.317142 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 02 20:54:51.572711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:51.572623 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 20:54:51.572711 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:51.572703 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 20:54:52.455483 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:52.455446 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:52.456223 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:52.455966 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:52.456857 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:52.456837 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 20:54:58.270786 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:58.270754 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:54:58.270786 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:58.270784 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:58.270786 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:58.270793 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:58.271362 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:54:58.270862 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:55:14.270842081 +0000 UTC m=+3997.877386576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:54:59.436272 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:54:59.436232 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:00.214226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.214193 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:55:00.219978 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.219956 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.269546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.269504 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dd5d5-openshift-default-dockercfg-zfjmp\"" Nov 02 20:55:00.296527 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296493 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.296771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296541 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t942h\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.296771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296569 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.296771 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296742 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.296959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296800 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.296959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.296938 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.297072 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.297039 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.297130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.297098 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.297187 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.297157 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.320926 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.320891 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:55:00.398394 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398358 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398413 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398441 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398522 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398560 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t942h\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398611 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398597 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.398921 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.398906 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399050 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399018 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399181 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399050 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399181 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399139 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399218 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399221 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399307 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399278 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.399506 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.399482 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.401409 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.401381 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.401578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.401561 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.453365 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.453330 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.454031 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.454005 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t942h\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\") pod \"e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.534417 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.534327 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:00.964440 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.964407 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:55:00.964784 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:55:00.964761 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82025e4_6b87_4eb2_a171_da812aa8e860.slice/crio-0727379fa3ecebd9f39f6a1c13ed970736b3e5550ca9cc98478b03943440eddb WatchSource:0}: Error finding container 0727379fa3ecebd9f39f6a1c13ed970736b3e5550ca9cc98478b03943440eddb: Status 404 returned error can't find the container with id 0727379fa3ecebd9f39f6a1c13ed970736b3e5550ca9cc98478b03943440eddb Nov 02 20:55:00.967705 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.967673 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:55:00.967803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.967765 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:55:00.967803 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:00.967798 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:55:01.726101 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:01.726064 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerStarted","Data":"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760"} Nov 02 20:55:01.726525 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:01.726117 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerStarted","Data":"0727379fa3ecebd9f39f6a1c13ed970736b3e5550ca9cc98478b03943440eddb"} Nov 02 20:55:01.838692 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:01.838643 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podStartSLOduration=1.838625925 podStartE2EDuration="1.838625925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:01.835891655 +0000 UTC m=+3985.442436153" watchObservedRunningTime="2026-11-02 20:55:01.838625925 +0000 UTC m=+3985.445170424" Nov 02 20:55:02.534518 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:02.534481 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:02.547546 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:02.547521 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:02.732567 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:02.732539 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:02.733597 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:02.733573 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:55:09.436359 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:09.436321 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:14.294106 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:14.294072 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:55:14.294106 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:14.294105 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:55:14.294106 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:14.294115 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:55:14.294583 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:14.294171 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:55:46.294152842 +0000 UTC m=+4029.900697321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:55:19.435461 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:19.435424 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:21.004306 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:21.004267 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:55:29.436415 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:29.436376 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:39.435824 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:39.435783 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:45.540196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:45.540156 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:46.324860 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:46.324828 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:55:46.324860 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:46.324858 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:55:46.324860 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:46.324867 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:55:46.325201 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:46.324934 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:56:50.324915384 +0000 UTC m=+4093.931459877 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:55:51.006170 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:51.006128 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:55:53.158724 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.158686 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:55:53.159199 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.158921 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" podUID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" containerName="maas-api" containerID="cri-o://3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7" gracePeriod=30 Nov 02 20:55:53.446584 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.446558 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:55:53.527788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.527742 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\") pod \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " Nov 02 20:55:53.527788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.527792 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") pod \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " Nov 02 20:55:53.528022 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.527901 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\" (UniqueName: \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\") pod \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " Nov 02 20:55:53.528128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.528101 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\") pod \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\" (UID: \"e46746f6-2fde-43d7-ae1a-bd16219f7bc1\") " Nov 02 20:55:53.528187 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.528168 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca" pod "e46746f6-2fde-43d7-ae1a-bd16219f7bc1" (UID: "e46746f6-2fde-43d7-ae1a-bd16219f7bc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:55:53.528780 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.528758 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:53.530042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.530010 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls" pod "e46746f6-2fde-43d7-ae1a-bd16219f7bc1" (UID: "e46746f6-2fde-43d7-ae1a-bd16219f7bc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:55:53.530151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.530097 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7" pod "e46746f6-2fde-43d7-ae1a-bd16219f7bc1" (UID: "e46746f6-2fde-43d7-ae1a-bd16219f7bc1"). InnerVolumeSpecName "kube-api-access-l6dx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:53.530151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.530120 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs" pod "e46746f6-2fde-43d7-ae1a-bd16219f7bc1" (UID: "e46746f6-2fde-43d7-ae1a-bd16219f7bc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:55:53.630807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.630770 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:53.630807 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.630806 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dx7\" (UniqueName: \"kubernetes.io/projected/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-kube-api-access-l6dx7\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:53.631313 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:53.631289 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46746f6-2fde-43d7-ae1a-bd16219f7bc1-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:54.066573 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.066541 5136 generic.go:356] "Generic (PLEG): container finished" podID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" containerID="3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7" exitCode=0 Nov 02 20:55:54.066796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.066593 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" Nov 02 20:55:54.066796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.066637 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" event={"ID":"e46746f6-2fde-43d7-ae1a-bd16219f7bc1","Type":"ContainerDied","Data":"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7"} Nov 02 20:55:54.066796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.066687 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l" event={"ID":"e46746f6-2fde-43d7-ae1a-bd16219f7bc1","Type":"ContainerDied","Data":"3b5c9cb42242f1ece0d492e3c32727b1c1db7c0788d9010543efa8ec7cb3a7d1"} Nov 02 20:55:54.066796 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.066713 5136 scope.go:122] "RemoveContainer" containerID="3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7" Nov 02 20:55:54.077833 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.077813 5136 scope.go:122] "RemoveContainer" containerID="3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7" Nov 02 20:55:54.078152 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:54.078132 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7\": container with ID starting with 3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7 not found: ID does not exist" containerID="3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7" Nov 02 20:55:54.078226 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.078164 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7"} err="failed to get container status \"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7\": rpc error: code = NotFound desc = could not find container \"3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7\": container with ID starting with 3e8b4b72668126b5c1f3e936def146663976ad5e538f5267522eba90fd8219f7 not found: ID does not exist" Nov 02 20:55:54.224865 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.224831 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:55:54.227548 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.227524 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-020389-6549b865f5-5qp5l"] Nov 02 20:55:54.548061 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.548020 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" path="/var/lib/kubelet/pods/e46746f6-2fde-43d7-ae1a-bd16219f7bc1/volumes" Nov 02 20:55:54.758245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.758211 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 20:55:54.758910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.758894 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" containerName="maas-api" Nov 02 20:55:54.758910 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.758909 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" containerName="maas-api" Nov 02 20:55:54.759054 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.759044 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46746f6-2fde-43d7-ae1a-bd16219f7bc1" containerName="maas-api" Nov 02 20:55:54.764048 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.764028 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.815277 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.815152 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 20:55:54.854767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.854694 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.854767 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.854768 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.855043 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.855010 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.855193 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.855166 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927td\" (UniqueName: \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.857954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.857932 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f3109b-serving-cert\"" Nov 02 20:55:54.956299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.956258 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.956299 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.956310 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.956588 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:54.956426 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-serving-cert: secret "maas-api-e2e-inf-a-f3109b-serving-cert" not found Nov 02 20:55:54.956588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.956460 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.956588 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:55:54.956499 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls podName:9e1dfc8a-4822-4962-b96e-b4d02cad7581 nodeName:}" failed. No retries permitted until 2026-11-02 20:55:55.456478016 +0000 UTC m=+4039.063022512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls") pod "maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" (UID: "9e1dfc8a-4822-4962-b96e-b4d02cad7581") : secret "maas-api-e2e-inf-a-f3109b-serving-cert" not found Nov 02 20:55:54.956588 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.956563 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927td\" (UniqueName: \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.957245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.957220 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:54.958886 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:54.958866 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:55.020013 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:55.016204 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927td\" (UniqueName: \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:55.464270 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:55.464235 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:55.466782 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:55.466761 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:55.540135 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:55.540087 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:55:55.677787 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:55.677722 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:56.046349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:56.046318 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 20:55:56.047230 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:55:56.047192 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1dfc8a_4822_4962_b96e_b4d02cad7581.slice/crio-7210cabef027f923e015379836ab482f1c4b04c46c74679e82449b7a4dd76a64 WatchSource:0}: Error finding container 7210cabef027f923e015379836ab482f1c4b04c46c74679e82449b7a4dd76a64: Status 404 returned error can't find the container with id 7210cabef027f923e015379836ab482f1c4b04c46c74679e82449b7a4dd76a64 Nov 02 20:55:56.048970 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:56.048951 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:55:56.083157 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:56.083125 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" event={"ID":"9e1dfc8a-4822-4962-b96e-b4d02cad7581","Type":"ContainerStarted","Data":"7210cabef027f923e015379836ab482f1c4b04c46c74679e82449b7a4dd76a64"} Nov 02 20:55:57.091012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:57.090918 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" event={"ID":"9e1dfc8a-4822-4962-b96e-b4d02cad7581","Type":"ContainerStarted","Data":"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3"} Nov 02 20:55:57.091012 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:57.090983 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:55:57.199819 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:55:57.199768 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" podStartSLOduration=2.409923066 podStartE2EDuration="3.19972305s" totalImagesPullingTime="789.799984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:57.198930416 +0000 UTC m=+4040.805474914" watchObservedRunningTime="2026-11-02 20:55:57.19972305 +0000 UTC m=+4040.806267548" Nov 02 20:56:00.476963 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.476931 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 20:56:00.481516 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.481488 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.531578 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.531542 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4530c5-openshift-default-dockercfg-6kwdh\"" Nov 02 20:56:00.585125 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.585087 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 20:56:00.641614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.641577 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.641614 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.641622 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.641957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.641816 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.641957 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.641890 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.645823 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.645784 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.645984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.645902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.645984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.645934 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.645984 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.645968 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnn5j\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.646130 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.646070 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.747909 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnn5j\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.747989 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748062 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748088 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748157 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748293 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748244 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748328 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748404 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748495 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748444 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748653 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748582 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748721 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748652 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748841 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748794 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.748900 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.748868 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.749070 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.749048 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.751108 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.751077 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.751532 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.751511 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.804124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.803940 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:00.805920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:00.805889 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnn5j\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\") pod \"e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:01.095517 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:01.095437 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:01.522281 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:01.517716 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 20:56:01.529105 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:01.529063 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:01.529230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:01.529153 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:01.529230 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:01.529195 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:02.134827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:02.134786 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerStarted","Data":"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a"} Nov 02 20:56:02.136998 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:02.134860 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerStarted","Data":"fe490b8b8bf987488c7ac39d4a58811a2fbb7121d07d47fe23dacc3f78c06519"} Nov 02 20:56:02.298560 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:02.298490 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podStartSLOduration=2.298467922 podStartE2EDuration="2.298467922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:02.293308682 +0000 UTC m=+4045.899853192" watchObservedRunningTime="2026-11-02 20:56:02.298467922 +0000 UTC m=+4045.905012421" Nov 02 20:56:03.104763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:03.100233 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:03.113763 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:03.111880 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:03.150788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:03.145028 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 20:56:03.177498 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:03.177457 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:03.178667 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:03.178645 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:56:05.540726 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:05.540691 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:15.540477 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:15.540439 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:21.012438 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:21.012362 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:56:25.540511 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:25.540474 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:28.873876 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.873839 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 20:56:28.878647 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.878621 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.929074 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.929042 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:56:28.929352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.929042 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:56:28.929352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.929069 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:56:28.929554 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.929084 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqzr\"" Nov 02 20:56:28.993756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.972705 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fl5h\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.993756 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.982572 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995005 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995178 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995261 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995385 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995411 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:28.996196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:28.995433 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113449 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fl5h\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113492 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113497 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113565 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113609 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113678 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113695 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.113761 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.113708 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.114451 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.114418 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.116536 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.116485 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.116697 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.116674 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.116792 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.116766 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.116896 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.116878 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.168231 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.168105 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.168720 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.168693 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fl5h\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7sv8\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.218254 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.218211 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:29.641235 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.641205 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 20:56:29.641445 ip-10-0-128-34 kubenswrapper[5136]: W1102 20:56:29.641421 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66b6b27_5cb5_42f1_9709_ee27ac95695a.slice/crio-5814138289ff7f131cbf69ff6d7c3318dcdd87e9445947280dc36ef1b67080b5 WatchSource:0}: Error finding container 5814138289ff7f131cbf69ff6d7c3318dcdd87e9445947280dc36ef1b67080b5: Status 404 returned error can't find the container with id 5814138289ff7f131cbf69ff6d7c3318dcdd87e9445947280dc36ef1b67080b5 Nov 02 20:56:29.644325 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.644288 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:29.644444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:29.644361 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:30.427044 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.426999 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" event={"ID":"c66b6b27-5cb5-42f1-9709-ee27ac95695a","Type":"ContainerStarted","Data":"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66"} Nov 02 20:56:30.427686 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.427665 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:30.427864 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.427846 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" event={"ID":"c66b6b27-5cb5-42f1-9709-ee27ac95695a","Type":"ContainerStarted","Data":"5814138289ff7f131cbf69ff6d7c3318dcdd87e9445947280dc36ef1b67080b5"} Nov 02 20:56:30.429023 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.428978 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-p7sv8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:56:30.429136 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.429027 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:56:30.629092 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:30.628999 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" podStartSLOduration=2.628983706 podStartE2EDuration="2.628983706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:30.585616052 +0000 UTC m=+4074.192160551" watchObservedRunningTime="2026-11-02 20:56:30.628983706 +0000 UTC m=+4074.235528203" Nov 02 20:56:31.435243 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:31.435202 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-p7sv8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:56:31.435671 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:31.435284 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:56:34.435770 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.435747 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 20:56:34.459827 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.459795 5136 generic.go:356] "Generic (PLEG): container finished" podID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerID="ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850" exitCode=0 Nov 02 20:56:34.460010 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.459867 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerDied","Data":"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850"} Nov 02 20:56:34.460358 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.460341 5136 scope.go:122] "RemoveContainer" containerID="ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850" Nov 02 20:56:34.462247 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.462212 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:34.462333 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.462298 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:34.462378 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:34.462347 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:35.468342 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:35.468301 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerStarted","Data":"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9"} Nov 02 20:56:35.540757 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:35.540709 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:36.451444 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:36.451400 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:56:36.453128 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:36.453098 5136 patch_prober.go:29] interesting pod/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 20:56:36.453262 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:36.453146 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 20:56:37.451352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:37.451316 5136 patch_prober.go:29] interesting pod/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 20:56:37.451779 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:37.451374 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 20:56:38.451371 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:38.451340 5136 patch_prober.go:29] interesting pod/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 20:56:38.451813 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:38.451393 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 20:56:39.406762 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:39.401845 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:56:39.451245 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:39.451206 5136 patch_prober.go:29] interesting pod/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 20:56:39.451421 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:39.451261 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 20:56:40.451335 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:40.451299 5136 patch_prober.go:29] interesting pod/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 20:56:40.451520 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:40.451363 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 20:56:41.456685 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:41.456650 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:56:41.457954 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:41.457929 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 20:56:45.540383 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:45.540339 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:50.369338 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:50.369305 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:56:50.369823 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:50.369801 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:56:50.369925 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:50.369828 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:56:50.369925 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:50.369917 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 20:58:52.369895442 +0000 UTC m=+4215.976439931 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:56:51.025416 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:56:51.025380 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:56:51.540613 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:51.540563 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:56:55.626652 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.626617 5136 generic.go:356] "Generic (PLEG): container finished" podID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerID="dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760" exitCode=0 Nov 02 20:56:55.627173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.626689 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerDied","Data":"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760"} Nov 02 20:56:55.627173 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.627164 5136 scope.go:122] "RemoveContainer" containerID="dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760" Nov 02 20:56:55.629068 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.629042 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:55.629146 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.629110 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:55.629184 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:55.629144 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:56:56.635020 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:56.634989 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerStarted","Data":"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2"} Nov 02 20:56:57.534718 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:57.534681 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:56:57.534968 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:57.534897 5136 patch_prober.go:29] interesting pod/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 20:56:57.534968 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:57.534950 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 20:56:58.534920 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:58.534886 5136 patch_prober.go:29] interesting pod/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 20:56:58.535334 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:58.534936 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 20:56:59.535617 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:59.535576 5136 patch_prober.go:29] interesting pod/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 20:56:59.536042 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:56:59.535634 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 20:57:00.535716 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:00.535679 5136 patch_prober.go:29] interesting pod/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 20:57:00.536177 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:00.535765 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 20:57:01.535523 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:01.535486 5136 patch_prober.go:29] interesting pod/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 02 20:57:01.535725 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:01.535549 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 02 20:57:01.540678 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:01.540648 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:02.547643 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:02.547613 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:57:02.548093 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:02.547974 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:57:02.548983 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:02.548960 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:57:11.540485 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:11.540446 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:21.012213 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:57:21.012169 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:57:21.540831 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:21.540790 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:31.541170 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:31.541129 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:41.540906 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:41.540860 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:51.006997 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:57:51.006906 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:57:51.541550 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:51.541502 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:57:56.544839 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:57:56.544797 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:06.545165 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:06.545125 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:16.545052 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:16.545012 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:18.179124 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.179091 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:18.179592 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.179143 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:18.205777 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.205721 5136 generic.go:356] "Generic (PLEG): container finished" podID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerID="058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a" exitCode=0 Nov 02 20:58:18.205987 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.205797 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerDied","Data":"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a"} Nov 02 20:58:18.206269 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.206251 5136 scope.go:122] "RemoveContainer" containerID="058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a" Nov 02 20:58:18.207860 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.207826 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:58:18.207932 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.207901 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:58:18.207981 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:18.207933 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:58:19.217798 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:19.217758 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerStarted","Data":"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862"} Nov 02 20:58:20.105755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:20.101178 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:58:20.105755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:20.102835 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:20.105755 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:20.102887 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:21.023546 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:21.023506 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:58:21.112759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:21.111882 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:21.112759 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:21.111948 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:22.100225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:22.100184 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:22.100996 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:22.100962 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:23.102323 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:23.102286 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:23.102959 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:23.102925 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:24.100252 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:24.100214 5136 patch_prober.go:29] interesting pod/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 20:58:24.100585 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:24.100553 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 20:58:25.100563 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:25.100529 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:58:25.101109 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:25.100908 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:58:25.101808 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:25.101790 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 20:58:26.545284 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:26.545249 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:36.544706 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:36.544670 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:46.544845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:46.544800 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:58:51.017901 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:51.017853 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:58:52.409250 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:52.409213 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 20:58:52.409250 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:52.409246 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:58:52.409250 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:52.409256 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:58:52.409755 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:58:52.409327 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 21:00:54.409310393 +0000 UTC m=+4338.015854869 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 20:58:56.544853 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:58:56.544811 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:02.540298 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:02.540257 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:12.540972 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:12.540931 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:21.006716 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:59:21.006635 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:59:22.540225 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:22.540188 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:32.540321 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:32.540272 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:37.172151 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:37.172105 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:59:37.172657 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:37.172377 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" containerID="cri-o://b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2" gracePeriod=30 Nov 02 20:59:42.456336 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.456307 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:59:42.540939 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.540901 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:59:42.559454 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559429 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559462 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559496 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559551 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559629 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559606 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559648 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559687 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559741 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.559917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559833 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:42.559917 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559886 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:42.560196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.559927 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\") pod \"a82025e4-6b87-4eb2-a171-da812aa8e860\" (UID: \"a82025e4-6b87-4eb2-a171-da812aa8e860\") " Nov 02 20:59:42.560196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560085 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:42.560196 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560103 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:42.560352 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560230 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:59:42.560564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560545 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.560564 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560564 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.560708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560574 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82025e4-6b87-4eb2-a171-da812aa8e860-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.560708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560591 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.560708 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.560605 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.561894 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.561869 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:59:42.562182 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.562166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "kube-api-access-t942h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:42.562341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.562315 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:42.562341 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.562316 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy" pod "a82025e4-6b87-4eb2-a171-da812aa8e860" (UID: "a82025e4-6b87-4eb2-a171-da812aa8e860"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:42.661788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.661745 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.661788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.661778 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.661788 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.661791 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82025e4-6b87-4eb2-a171-da812aa8e860-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.662057 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.661804 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t942h\" (UniqueName: \"kubernetes.io/projected/a82025e4-6b87-4eb2-a171-da812aa8e860-kube-api-access-t942h\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:42.797842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.797711 5136 generic.go:356] "Generic (PLEG): container finished" podID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerID="b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2" exitCode=0 Nov 02 20:59:42.797842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.797798 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" Nov 02 20:59:42.797842 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.797802 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerDied","Data":"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2"} Nov 02 20:59:42.798150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.797869 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l" event={"ID":"a82025e4-6b87-4eb2-a171-da812aa8e860","Type":"ContainerDied","Data":"0727379fa3ecebd9f39f6a1c13ed970736b3e5550ca9cc98478b03943440eddb"} Nov 02 20:59:42.798150 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.797898 5136 scope.go:122] "RemoveContainer" containerID="b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2" Nov 02 20:59:42.809557 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.809538 5136 scope.go:122] "RemoveContainer" containerID="dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760" Nov 02 20:59:42.819393 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.819352 5136 scope.go:122] "RemoveContainer" containerID="b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2" Nov 02 20:59:42.819706 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:59:42.819683 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2\": container with ID starting with b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2 not found: ID does not exist" containerID="b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2" Nov 02 20:59:42.819845 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.819717 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2"} err="failed to get container status \"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2\": rpc error: code = NotFound desc = could not find container \"b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2\": container with ID starting with b5eae543faa3732af2f3e23f908d83f514ffb0181f9a084812f18a3413139bf2 not found: ID does not exist" Nov 02 20:59:42.819935 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.819850 5136 scope.go:122] "RemoveContainer" containerID="dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760" Nov 02 20:59:42.820130 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:59:42.820106 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760\": container with ID starting with dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760 not found: ID does not exist" containerID="dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760" Nov 02 20:59:42.820178 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.820136 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760"} err="failed to get container status \"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760\": rpc error: code = NotFound desc = could not find container \"dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760\": container with ID starting with dca29bc3437252ef70cf836ae33bbd6cf1bbb81ce5e2dcd5b59d2248d81e6760 not found: ID does not exist" Nov 02 20:59:42.955385 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.955345 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:59:42.958349 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:42.958322 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dd5d5-openshift-default-6fcf5946b5-lzt7l"] Nov 02 20:59:44.548195 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:44.548163 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" path="/var/lib/kubelet/pods/a82025e4-6b87-4eb2-a171-da812aa8e860/volumes" Nov 02 20:59:51.005424 ip-10-0-128-34 kubenswrapper[5136]: E1102 20:59:51.005387 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 20:59:52.540884 ip-10-0-128-34 kubenswrapper[5136]: I1102 20:59:52.540848 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:00.225501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.225457 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:00.226377 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.226354 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:00:00.226481 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.226378 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:00:00.226604 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.226589 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:00:00.226665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.226609 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:00:00.229718 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.229696 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.319824 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.319790 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:00:00.327955 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.327911 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42wx\" (UniqueName: \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.328265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.328234 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.409233 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.409194 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:00.430039 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.430004 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.430256 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.430114 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42wx\" (UniqueName: \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.430753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.430698 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.484054 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.483977 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42wx\" (UniqueName: \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.543280 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.543248 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:00.910939 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.910907 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:00.911593 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:00:00.911568 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4d711d_52b0_4e94_811a_6e70e2a6a633.slice/crio-dc131f7ff5910b67bffd199b1f84542a1f07b0a4711de8618c8346eaf9ac1d54 WatchSource:0}: Error finding container dc131f7ff5910b67bffd199b1f84542a1f07b0a4711de8618c8346eaf9ac1d54: Status 404 returned error can't find the container with id dc131f7ff5910b67bffd199b1f84542a1f07b0a4711de8618c8346eaf9ac1d54 Nov 02 21:00:00.926676 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:00.926645 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerStarted","Data":"dc131f7ff5910b67bffd199b1f84542a1f07b0a4711de8618c8346eaf9ac1d54"} Nov 02 21:00:02.541515 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:02.541467 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:08.998181 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:08.998082 5136 generic.go:356] "Generic (PLEG): container finished" podID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerID="23d1134d3ae6fd00da7e95133c530036011599838ea0cb49a056d66faabd66cf" exitCode=60 Nov 02 21:00:08.998724 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:08.998323 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerDied","Data":"23d1134d3ae6fd00da7e95133c530036011599838ea0cb49a056d66faabd66cf"} Nov 02 21:00:08.999029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:08.998933 5136 scope.go:122] "RemoveContainer" containerID="23d1134d3ae6fd00da7e95133c530036011599838ea0cb49a056d66faabd66cf" Nov 02 21:00:10.007897 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:10.007870 5136 generic.go:356] "Generic (PLEG): container finished" podID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerID="bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7" exitCode=60 Nov 02 21:00:10.008305 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:10.007946 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerDied","Data":"bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7"} Nov 02 21:00:10.008305 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:10.008004 5136 scope.go:122] "RemoveContainer" containerID="23d1134d3ae6fd00da7e95133c530036011599838ea0cb49a056d66faabd66cf" Nov 02 21:00:10.008409 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:10.008304 5136 scope.go:122] "RemoveContainer" containerID="bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7" Nov 02 21:00:10.008653 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:10.008630 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd_odh-ai-gateway-infra(7f4d711d-52b0-4e94-811a-6e70e2a6a633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" Nov 02 21:00:11.016261 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:11.016232 5136 scope.go:122] "RemoveContainer" containerID="bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7" Nov 02 21:00:11.016677 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:11.016569 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd_odh-ai-gateway-infra(7f4d711d-52b0-4e94-811a-6e70e2a6a633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" Nov 02 21:00:12.541179 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:12.541138 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:21.006705 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:21.006663 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:00:22.540935 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:22.540900 5136 scope.go:122] "RemoveContainer" containerID="bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7" Nov 02 21:00:22.544140 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:22.544091 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:23.102424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:23.102322 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerStarted","Data":"59ad9991c9266472ac85b3ead5a07b41d45667262fd4b3fdfcbe80a6e7f384d9"} Nov 02 21:00:23.216618 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:23.215801 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" podStartSLOduration=16.246971408 podStartE2EDuration="23.215781764s" totalImagesPullingTime="6.968810356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:23.211880449 +0000 UTC m=+4306.818424951" watchObservedRunningTime="2026-11-02 21:00:23.215781764 +0000 UTC m=+4306.822326263" Nov 02 21:00:24.173890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:24.173845 5136 generic.go:356] "Generic (PLEG): container finished" podID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerID="59ad9991c9266472ac85b3ead5a07b41d45667262fd4b3fdfcbe80a6e7f384d9" exitCode=60 Nov 02 21:00:24.174424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:24.173947 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerDied","Data":"59ad9991c9266472ac85b3ead5a07b41d45667262fd4b3fdfcbe80a6e7f384d9"} Nov 02 21:00:24.174424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:24.174011 5136 scope.go:122] "RemoveContainer" containerID="bde631cc707876a47a9685c3aef8494fc3a8d1db28f1d7a1afd68582d02ea8c7" Nov 02 21:00:24.174854 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:24.174829 5136 scope.go:122] "RemoveContainer" containerID="59ad9991c9266472ac85b3ead5a07b41d45667262fd4b3fdfcbe80a6e7f384d9" Nov 02 21:00:24.176442 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:24.176413 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd_odh-ai-gateway-infra(7f4d711d-52b0-4e94-811a-6e70e2a6a633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" Nov 02 21:00:24.257537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:24.257176 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:25.400770 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.400744 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:25.548942 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.548899 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\" (UniqueName: \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\") pod \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " Nov 02 21:00:25.549127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.549010 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\") pod \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\" (UID: \"7f4d711d-52b0-4e94-811a-6e70e2a6a633\") " Nov 02 21:00:25.549459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.549429 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca" pod "7f4d711d-52b0-4e94-811a-6e70e2a6a633" (UID: "7f4d711d-52b0-4e94-811a-6e70e2a6a633"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:25.549675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.549654 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4d711d-52b0-4e94-811a-6e70e2a6a633-maas-api-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:25.551320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.551297 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx" pod "7f4d711d-52b0-4e94-811a-6e70e2a6a633" (UID: "7f4d711d-52b0-4e94-811a-6e70e2a6a633"). InnerVolumeSpecName "kube-api-access-v42wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:25.651800 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:25.651765 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42wx\" (UniqueName: \"kubernetes.io/projected/7f4d711d-52b0-4e94-811a-6e70e2a6a633-kube-api-access-v42wx\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:26.190630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.190594 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" Nov 02 21:00:26.190630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.190620 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd" event={"ID":"7f4d711d-52b0-4e94-811a-6e70e2a6a633","Type":"ContainerDied","Data":"dc131f7ff5910b67bffd199b1f84542a1f07b0a4711de8618c8346eaf9ac1d54"} Nov 02 21:00:26.190936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.190697 5136 scope.go:122] "RemoveContainer" containerID="59ad9991c9266472ac85b3ead5a07b41d45667262fd4b3fdfcbe80a6e7f384d9" Nov 02 21:00:26.349647 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.349604 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:26.397075 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.397037 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894220-fvnbd"] Nov 02 21:00:26.547793 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:26.547760 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" path="/var/lib/kubelet/pods/7f4d711d-52b0-4e94-811a-6e70e2a6a633/volumes" Nov 02 21:00:31.540613 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:31.540573 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:37.310173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:37.310135 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 21:00:37.411089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:37.411056 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 21:00:41.540440 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:41.540395 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:42.311418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.311346 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" podUID="85b94f48-d953-458f-be3b-2505a67c7538" containerName="payload-processing" containerID="cri-o://2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d" gracePeriod=25 Nov 02 21:00:42.411716 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.411650 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" podUID="772c9378-13ae-44ec-93a1-24c271e49713" containerName="payload-pre-processing" containerID="cri-o://7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4" gracePeriod=25 Nov 02 21:00:42.596832 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.596759 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 21:00:42.677652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.677629 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 21:00:42.706148 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.706117 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\") pod \"85b94f48-d953-458f-be3b-2505a67c7538\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " Nov 02 21:00:42.706302 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.706185 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\" (UniqueName: \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\") pod \"85b94f48-d953-458f-be3b-2505a67c7538\" (UID: \"85b94f48-d953-458f-be3b-2505a67c7538\") " Nov 02 21:00:42.706652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.706619 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume" pod "85b94f48-d953-458f-be3b-2505a67c7538" (UID: "85b94f48-d953-458f-be3b-2505a67c7538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:42.706965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.706931 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b94f48-d953-458f-be3b-2505a67c7538-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:42.709197 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.709170 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j" pod "85b94f48-d953-458f-be3b-2505a67c7538" (UID: "85b94f48-d953-458f-be3b-2505a67c7538"). InnerVolumeSpecName "kube-api-access-87f9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:42.807724 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.807682 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\") pod \"772c9378-13ae-44ec-93a1-24c271e49713\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " Nov 02 21:00:42.807963 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.807901 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\" (UniqueName: \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\") pod \"772c9378-13ae-44ec-93a1-24c271e49713\" (UID: \"772c9378-13ae-44ec-93a1-24c271e49713\") " Nov 02 21:00:42.808063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.808040 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume" pod "772c9378-13ae-44ec-93a1-24c271e49713" (UID: "772c9378-13ae-44ec-93a1-24c271e49713"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:42.808387 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.808367 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772c9378-13ae-44ec-93a1-24c271e49713-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:42.808459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.808392 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87f9j\" (UniqueName: \"kubernetes.io/projected/85b94f48-d953-458f-be3b-2505a67c7538-kube-api-access-87f9j\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:42.810078 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.810055 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w" pod "772c9378-13ae-44ec-93a1-24c271e49713" (UID: "772c9378-13ae-44ec-93a1-24c271e49713"). InnerVolumeSpecName "kube-api-access-7zz9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:42.909967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:42.909889 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz9w\" (UniqueName: \"kubernetes.io/projected/772c9378-13ae-44ec-93a1-24c271e49713-kube-api-access-7zz9w\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:43.306152 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.306121 5136 generic.go:356] "Generic (PLEG): container finished" podID="85b94f48-d953-458f-be3b-2505a67c7538" containerID="2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d" exitCode=0 Nov 02 21:00:43.306365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.306174 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" Nov 02 21:00:43.306365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.306213 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" event={"ID":"85b94f48-d953-458f-be3b-2505a67c7538","Type":"ContainerDied","Data":"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d"} Nov 02 21:00:43.306365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.306263 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m" event={"ID":"85b94f48-d953-458f-be3b-2505a67c7538","Type":"ContainerDied","Data":"e95d876204997eac1c09dccf439352eb06f9947401f94b0428b3d0ad9d1799ed"} Nov 02 21:00:43.306365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.306289 5136 scope.go:122] "RemoveContainer" containerID="2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d" Nov 02 21:00:43.307782 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.307761 5136 generic.go:356] "Generic (PLEG): container finished" podID="772c9378-13ae-44ec-93a1-24c271e49713" containerID="7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4" exitCode=0 Nov 02 21:00:43.307914 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.307789 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" event={"ID":"772c9378-13ae-44ec-93a1-24c271e49713","Type":"ContainerDied","Data":"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4"} Nov 02 21:00:43.307914 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.307797 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" Nov 02 21:00:43.307914 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.307825 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5" event={"ID":"772c9378-13ae-44ec-93a1-24c271e49713","Type":"ContainerDied","Data":"a432f493a7801fd05950fcc8c085907eb90513fa92fa123f3b7c862a73b028be"} Nov 02 21:00:43.318460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.318072 5136 scope.go:122] "RemoveContainer" containerID="2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d" Nov 02 21:00:43.318460 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:43.318440 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d\": container with ID starting with 2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d not found: ID does not exist" containerID="2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d" Nov 02 21:00:43.318558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.318468 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d"} err="failed to get container status \"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d\": rpc error: code = NotFound desc = could not find container \"2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d\": container with ID starting with 2c225a8fd6a4c2251bb54e74a7a56c29485ab96a77a4fb552ac5bc35834a9b7d not found: ID does not exist" Nov 02 21:00:43.318558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.318494 5136 scope.go:122] "RemoveContainer" containerID="7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4" Nov 02 21:00:43.339359 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.339336 5136 scope.go:122] "RemoveContainer" containerID="7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4" Nov 02 21:00:43.339613 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:43.339593 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4\": container with ID starting with 7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4 not found: ID does not exist" containerID="7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4" Nov 02 21:00:43.339663 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.339620 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4"} err="failed to get container status \"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4\": rpc error: code = NotFound desc = could not find container \"7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4\": container with ID starting with 7db8d29e1c38170eecc04509ab396f75beedcdfa3d4b0ae5a5e996be5a0034e4 not found: ID does not exist" Nov 02 21:00:43.461174 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.461139 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 21:00:43.466702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.466675 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f97409-b4d8b99b7-xrx6m"] Nov 02 21:00:43.570132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.570057 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 21:00:43.572986 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:43.572962 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f97409-97d79f8b7-zr9t5"] Nov 02 21:00:44.420186 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:44.420152 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 21:00:44.522313 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:44.522281 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 21:00:44.548121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:44.548086 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772c9378-13ae-44ec-93a1-24c271e49713" path="/var/lib/kubelet/pods/772c9378-13ae-44ec-93a1-24c271e49713/volumes" Nov 02 21:00:44.548525 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:44.548511 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b94f48-d953-458f-be3b-2505a67c7538" path="/var/lib/kubelet/pods/85b94f48-d953-458f-be3b-2505a67c7538/volumes" Nov 02 21:00:49.420649 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.420569 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" podUID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" containerName="payload-processing" containerID="cri-o://d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50" gracePeriod=25 Nov 02 21:00:49.522801 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.522760 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" podUID="ef68c87a-2391-4e67-a202-7804a6c1f380" containerName="payload-pre-processing" containerID="cri-o://447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723" gracePeriod=25 Nov 02 21:00:49.703973 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.703944 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 21:00:49.741955 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.741921 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\" (UniqueName: \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\") pod \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " Nov 02 21:00:49.742089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.742053 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\") pod \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\" (UID: \"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432\") " Nov 02 21:00:49.742462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.742432 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume" pod "aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" (UID: "aa8a4baf-a24b-4cb5-a1d4-55c08df2a432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:49.742858 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.742836 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:49.745489 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.745461 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck" pod "aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" (UID: "aa8a4baf-a24b-4cb5-a1d4-55c08df2a432"). InnerVolumeSpecName "kube-api-access-sj6ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:49.799055 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.799027 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 21:00:49.843527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.843492 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\") pod \"ef68c87a-2391-4e67-a202-7804a6c1f380\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " Nov 02 21:00:49.843689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.843587 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\" (UniqueName: \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\") pod \"ef68c87a-2391-4e67-a202-7804a6c1f380\" (UID: \"ef68c87a-2391-4e67-a202-7804a6c1f380\") " Nov 02 21:00:49.843968 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.843942 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume" pod "ef68c87a-2391-4e67-a202-7804a6c1f380" (UID: "ef68c87a-2391-4e67-a202-7804a6c1f380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:49.844253 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.844230 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6ck\" (UniqueName: \"kubernetes.io/projected/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432-kube-api-access-sj6ck\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:49.844338 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.844257 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68c87a-2391-4e67-a202-7804a6c1f380-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:49.845855 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.845828 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7" pod "ef68c87a-2391-4e67-a202-7804a6c1f380" (UID: "ef68c87a-2391-4e67-a202-7804a6c1f380"). InnerVolumeSpecName "kube-api-access-4g8v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:49.945878 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:49.945804 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8v7\" (UniqueName: \"kubernetes.io/projected/ef68c87a-2391-4e67-a202-7804a6c1f380-kube-api-access-4g8v7\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:50.357454 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.357374 5136 generic.go:356] "Generic (PLEG): container finished" podID="ef68c87a-2391-4e67-a202-7804a6c1f380" containerID="447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723" exitCode=0 Nov 02 21:00:50.357454 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.357425 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" Nov 02 21:00:50.357667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.357463 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" event={"ID":"ef68c87a-2391-4e67-a202-7804a6c1f380","Type":"ContainerDied","Data":"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723"} Nov 02 21:00:50.357667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.357521 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s" event={"ID":"ef68c87a-2391-4e67-a202-7804a6c1f380","Type":"ContainerDied","Data":"d0570839f9fa0cf691acd5f7e6286c3261a0442f64881a682689fab23340ed7d"} Nov 02 21:00:50.357667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.357553 5136 scope.go:122] "RemoveContainer" containerID="447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723" Nov 02 21:00:50.358862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.358838 5136 generic.go:356] "Generic (PLEG): container finished" podID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" containerID="d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50" exitCode=0 Nov 02 21:00:50.359020 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.358876 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" Nov 02 21:00:50.359020 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.358914 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" event={"ID":"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432","Type":"ContainerDied","Data":"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50"} Nov 02 21:00:50.359020 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.358970 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt" event={"ID":"aa8a4baf-a24b-4cb5-a1d4-55c08df2a432","Type":"ContainerDied","Data":"5b8e0bc4a2052531b9e58334d0a76704c9cb26c3000049e6f699b982b5b996f4"} Nov 02 21:00:50.369488 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.369466 5136 scope.go:122] "RemoveContainer" containerID="447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723" Nov 02 21:00:50.369853 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:50.369752 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723\": container with ID starting with 447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723 not found: ID does not exist" containerID="447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723" Nov 02 21:00:50.369853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.369777 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723"} err="failed to get container status \"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723\": rpc error: code = NotFound desc = could not find container \"447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723\": container with ID starting with 447eb261add04090f65b0c851c2a670fa249d9d7192c44b774323b50886d7723 not found: ID does not exist" Nov 02 21:00:50.369853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.369798 5136 scope.go:122] "RemoveContainer" containerID="d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50" Nov 02 21:00:50.378466 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.378447 5136 scope.go:122] "RemoveContainer" containerID="d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50" Nov 02 21:00:50.378777 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:50.378757 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50\": container with ID starting with d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50 not found: ID does not exist" containerID="d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50" Nov 02 21:00:50.378853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.378783 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50"} err="failed to get container status \"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50\": rpc error: code = NotFound desc = could not find container \"d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50\": container with ID starting with d7165fefad6335b5a74905b8588c91fd08c39eab305b7651b9e2714eedacad50 not found: ID does not exist" Nov 02 21:00:50.522003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.521966 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 21:00:50.523702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.523670 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3109b-6cd644b56c-6kpbt"] Nov 02 21:00:50.546526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.546501 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" path="/var/lib/kubelet/pods/aa8a4baf-a24b-4cb5-a1d4-55c08df2a432/volumes" Nov 02 21:00:50.620849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.620779 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 21:00:50.623813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:50.623786 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3109b-7dd9589bdc-l669s"] Nov 02 21:00:51.005463 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:51.005426 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:00:51.540350 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:51.540317 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:00:52.547250 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:00:52.547207 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef68c87a-2391-4e67-a202-7804a6c1f380" path="/var/lib/kubelet/pods/ef68c87a-2391-4e67-a202-7804a6c1f380/volumes" Nov 02 21:00:54.501975 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:54.501941 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2da8cddd53dd: secret "agc-cred-2da8cddd53dd" not found Nov 02 21:00:54.501975 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:54.501976 5136 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 21:00:54.502460 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:54.501988 5136 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d68449d5-wszs7: [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 21:00:54.502460 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:00:54.502059 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials podName:a364c385-934d-4d00-ac4c-4c22ff3c4656 nodeName:}" failed. No retries permitted until 2026-11-02 21:02:56.502043475 +0000 UTC m=+4460.108587972 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials") pod "payload-processing-external-model-7d68449d5-wszs7" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656") : [secret "agc-cred-2da8cddd53dd" not found, secret "agc-cred-c233618dc3e5" not found] Nov 02 21:01:01.540869 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:01.540837 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:01:05.852398 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:05.852360 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 21:01:05.854073 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:05.854041 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:01:05.899893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:05.899860 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 21:01:10.853399 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:10.853357 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" containerID="cri-o://fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35" gracePeriod=25 Nov 02 21:01:10.901111 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:10.901074 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerName="payload-processing" containerID="cri-o://699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c" gracePeriod=25 Nov 02 21:01:10.955441 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:01:10.955421 5136 logging.go:55] [core] [Channel #6890 SubChannel #6891]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.178:50052", ServerName: "10.133.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.178:50052: connect: connection refused" Nov 02 21:01:11.167601 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.167572 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 21:01:11.207169 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.207147 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 21:01:11.236625 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.236595 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\") pod \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " Nov 02 21:01:11.236812 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.236721 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\") pod \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " Nov 02 21:01:11.236812 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.236781 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\") pod \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " Nov 02 21:01:11.236910 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.236832 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\") pod \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\" (UID: \"4e4429f2-5ab9-4205-9ff6-5828958c1c54\") " Nov 02 21:01:11.237051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.237027 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume" pod "4e4429f2-5ab9-4205-9ff6-5828958c1c54" (UID: "4e4429f2-5ab9-4205-9ff6-5828958c1c54"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:11.237103 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.237057 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume" pod "4e4429f2-5ab9-4205-9ff6-5828958c1c54" (UID: "4e4429f2-5ab9-4205-9ff6-5828958c1c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:11.237334 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.237316 5136 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-routing-overlay-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.237424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.237334 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4429f2-5ab9-4205-9ff6-5828958c1c54-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.239127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.239103 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials" pod "4e4429f2-5ab9-4205-9ff6-5828958c1c54" (UID: "4e4429f2-5ab9-4205-9ff6-5828958c1c54"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:11.239217 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.239135 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd" pod "4e4429f2-5ab9-4205-9ff6-5828958c1c54" (UID: "4e4429f2-5ab9-4205-9ff6-5828958c1c54"). InnerVolumeSpecName "kube-api-access-wnwwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:11.338765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.338716 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\") pod \"a364c385-934d-4d00-ac4c-4c22ff3c4656\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " Nov 02 21:01:11.338955 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.338835 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\") pod \"a364c385-934d-4d00-ac4c-4c22ff3c4656\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " Nov 02 21:01:11.338955 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.338863 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\") pod \"a364c385-934d-4d00-ac4c-4c22ff3c4656\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " Nov 02 21:01:11.339100 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.338998 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\") pod \"a364c385-934d-4d00-ac4c-4c22ff3c4656\" (UID: \"a364c385-934d-4d00-ac4c-4c22ff3c4656\") " Nov 02 21:01:11.339243 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339218 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume" pod "a364c385-934d-4d00-ac4c-4c22ff3c4656" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:11.339349 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339258 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume" pod "a364c385-934d-4d00-ac4c-4c22ff3c4656" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:11.339528 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339507 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.339570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339531 5136 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a364c385-934d-4d00-ac4c-4c22ff3c4656-routing-overlay-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.339570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339541 5136 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-provider-credentials\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.339570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.339554 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnwwd\" (UniqueName: \"kubernetes.io/projected/4e4429f2-5ab9-4205-9ff6-5828958c1c54-kube-api-access-wnwwd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.341144 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.341122 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq" pod "a364c385-934d-4d00-ac4c-4c22ff3c4656" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656"). InnerVolumeSpecName "kube-api-access-blmzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:11.341210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.341173 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials" pod "a364c385-934d-4d00-ac4c-4c22ff3c4656" (UID: "a364c385-934d-4d00-ac4c-4c22ff3c4656"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:11.441245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.441148 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmzq\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-kube-api-access-blmzq\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.441245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.441179 5136 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a364c385-934d-4d00-ac4c-4c22ff3c4656-provider-credentials\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:11.496934 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.496898 5136 generic.go:356] "Generic (PLEG): container finished" podID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerID="699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c" exitCode=0 Nov 02 21:01:11.497131 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.496947 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" Nov 02 21:01:11.497131 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.496989 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" event={"ID":"a364c385-934d-4d00-ac4c-4c22ff3c4656","Type":"ContainerDied","Data":"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c"} Nov 02 21:01:11.497131 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.497045 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" event={"ID":"a364c385-934d-4d00-ac4c-4c22ff3c4656","Type":"ContainerDied","Data":"1a8b13ac7d4055c92468c043e5e3b7035e46e81b63a16572cac54cb80bc5a29d"} Nov 02 21:01:11.497131 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.497073 5136 scope.go:122] "RemoveContainer" containerID="699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c" Nov 02 21:01:11.498606 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.498574 5136 generic.go:356] "Generic (PLEG): container finished" podID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerID="fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35" exitCode=0 Nov 02 21:01:11.498753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.498619 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" Nov 02 21:01:11.498753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.498626 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" event={"ID":"4e4429f2-5ab9-4205-9ff6-5828958c1c54","Type":"ContainerDied","Data":"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35"} Nov 02 21:01:11.498753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.498663 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb" event={"ID":"4e4429f2-5ab9-4205-9ff6-5828958c1c54","Type":"ContainerDied","Data":"dde3c195d6eccd34bf0d143f474d1ad95ad2da5f0184db9f8dbca7ae44910d38"} Nov 02 21:01:11.507315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.507294 5136 scope.go:122] "RemoveContainer" containerID="699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c" Nov 02 21:01:11.507612 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:11.507592 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c\": container with ID starting with 699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c not found: ID does not exist" containerID="699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c" Nov 02 21:01:11.507711 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.507625 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c"} err="failed to get container status \"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c\": rpc error: code = NotFound desc = could not find container \"699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c\": container with ID starting with 699ab0c1ca4e7eecf2dbf2682814406e01aeefe6ac20b9f02777914541b0bf0c not found: ID does not exist" Nov 02 21:01:11.507711 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.507647 5136 scope.go:122] "RemoveContainer" containerID="fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35" Nov 02 21:01:11.516170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.516151 5136 scope.go:122] "RemoveContainer" containerID="fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35" Nov 02 21:01:11.516443 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:11.516425 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35\": container with ID starting with fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35 not found: ID does not exist" containerID="fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35" Nov 02 21:01:11.516494 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.516449 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35"} err="failed to get container status \"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35\": rpc error: code = NotFound desc = could not find container \"fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35\": container with ID starting with fac7f3378649799d1fa20679cadf11f04f728edd59b7d7a4aa2ae1bc878daf35 not found: ID does not exist" Nov 02 21:01:11.657035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.656989 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 21:01:11.661369 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.661342 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fcf8d6949-lqfjb"] Nov 02 21:01:11.771417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.771373 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 21:01:11.778389 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.778357 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d68449d5-wszs7"] Nov 02 21:01:11.956046 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:11.956010 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7d68449d5-wszs7" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.178:50052\" within 1s: context deadline exceeded" Nov 02 21:01:12.546358 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:12.546328 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" path="/var/lib/kubelet/pods/4e4429f2-5ab9-4205-9ff6-5828958c1c54/volumes" Nov 02 21:01:12.546781 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:12.546767 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" path="/var/lib/kubelet/pods/a364c385-934d-4d00-ac4c-4c22ff3c4656/volumes" Nov 02 21:01:21.005237 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:21.005204 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:01:46.166888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.166850 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 21:01:46.167373 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.167053 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" containerID="cri-o://f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66" gracePeriod=30 Nov 02 21:01:46.537793 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.536666 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668043 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668109 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668153 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668239 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668281 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668317 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.671396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.668351 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\") pod \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\" (UID: \"c66b6b27-5cb5-42f1-9709-ee27ac95695a\") " Nov 02 21:01:46.675074 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.675037 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:46.679850 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.679824 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:46.679985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.679901 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:46.680166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.680121 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:46.680166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.680150 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:46.680166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.680150 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:46.680332 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.680172 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h" pod "c66b6b27-5cb5-42f1-9709-ee27ac95695a" (UID: "c66b6b27-5cb5-42f1-9709-ee27ac95695a"). InnerVolumeSpecName "kube-api-access-7fl5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.773948 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-ca-configmap\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.773986 5136 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-csr-dns-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.774005 5136 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-cacerts\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.774019 5136 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66b6b27-5cb5-42f1-9709-ee27ac95695a-local-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.774032 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.774045 5136 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c66b6b27-5cb5-42f1-9709-ee27ac95695a-istio-kubeconfig\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.778355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.774059 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fl5h\" (UniqueName: \"kubernetes.io/projected/c66b6b27-5cb5-42f1-9709-ee27ac95695a-kube-api-access-7fl5h\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:46.808749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.789470 5136 generic.go:356] "Generic (PLEG): container finished" podID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerID="f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66" exitCode=0 Nov 02 21:01:46.808749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.789565 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" event={"ID":"c66b6b27-5cb5-42f1-9709-ee27ac95695a","Type":"ContainerDied","Data":"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66"} Nov 02 21:01:46.808749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.789617 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" event={"ID":"c66b6b27-5cb5-42f1-9709-ee27ac95695a","Type":"ContainerDied","Data":"5814138289ff7f131cbf69ff6d7c3318dcdd87e9445947280dc36ef1b67080b5"} Nov 02 21:01:46.808749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.789644 5136 scope.go:122] "RemoveContainer" containerID="f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66" Nov 02 21:01:46.808749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.789836 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" Nov 02 21:01:46.871198 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.871177 5136 scope.go:122] "RemoveContainer" containerID="f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66" Nov 02 21:01:46.871500 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:46.871481 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66\": container with ID starting with f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66 not found: ID does not exist" containerID="f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66" Nov 02 21:01:46.871557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:46.871511 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66"} err="failed to get container status \"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66\": rpc error: code = NotFound desc = could not find container \"f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66\": container with ID starting with f0827bbbc3bacbed4400d42cb1c3df94c61ab34b090ade12ccb3b6f2d277ec66 not found: ID does not exist" Nov 02 21:01:47.009832 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:47.009789 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 21:01:47.017682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:47.016616 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8"] Nov 02 21:01:48.547744 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:48.547698 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" path="/var/lib/kubelet/pods/c66b6b27-5cb5-42f1-9709-ee27ac95695a/volumes" Nov 02 21:01:50.892114 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:50.892080 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 21:01:50.892523 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:50.892334 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" containerID="cri-o://b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9" gracePeriod=30 Nov 02 21:01:51.054494 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:51.054456 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:01:51.128421 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.128388 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 21:01:51.128627 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.128604 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" podUID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" containerName="maas-api" containerID="cri-o://2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3" gracePeriod=30 Nov 02 21:01:51.421575 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.421480 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 21:01:51.434680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.434643 5136 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-p7sv8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.190:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:01:51.434856 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.434710 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7sv8" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.190:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:01:51.440219 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440195 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\") pod \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " Nov 02 21:01:51.440348 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440328 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\") pod \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " Nov 02 21:01:51.440420 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440362 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\" (UniqueName: \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\") pod \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " Nov 02 21:01:51.440489 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440416 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") pod \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\" (UID: \"9e1dfc8a-4822-4962-b96e-b4d02cad7581\") " Nov 02 21:01:51.440785 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440723 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca" pod "9e1dfc8a-4822-4962-b96e-b4d02cad7581" (UID: "9e1dfc8a-4822-4962-b96e-b4d02cad7581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:51.440928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.440910 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1dfc8a-4822-4962-b96e-b4d02cad7581-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:51.442598 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.442571 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td" pod "9e1dfc8a-4822-4962-b96e-b4d02cad7581" (UID: "9e1dfc8a-4822-4962-b96e-b4d02cad7581"). InnerVolumeSpecName "kube-api-access-927td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:51.442928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.442902 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs" pod "9e1dfc8a-4822-4962-b96e-b4d02cad7581" (UID: "9e1dfc8a-4822-4962-b96e-b4d02cad7581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:51.443024 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.442932 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls" pod "9e1dfc8a-4822-4962-b96e-b4d02cad7581" (UID: "9e1dfc8a-4822-4962-b96e-b4d02cad7581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:51.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.564185 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927td\" (UniqueName: \"kubernetes.io/projected/9e1dfc8a-4822-4962-b96e-b4d02cad7581-kube-api-access-927td\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:51.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.564219 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:51.566317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.564235 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1dfc8a-4822-4962-b96e-b4d02cad7581-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:51.869093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.869058 5136 generic.go:356] "Generic (PLEG): container finished" podID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" containerID="2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3" exitCode=0 Nov 02 21:01:51.869273 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.869102 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" Nov 02 21:01:51.869273 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.869119 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" event={"ID":"9e1dfc8a-4822-4962-b96e-b4d02cad7581","Type":"ContainerDied","Data":"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3"} Nov 02 21:01:51.869273 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.869167 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj" event={"ID":"9e1dfc8a-4822-4962-b96e-b4d02cad7581","Type":"ContainerDied","Data":"7210cabef027f923e015379836ab482f1c4b04c46c74679e82449b7a4dd76a64"} Nov 02 21:01:51.869273 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.869190 5136 scope.go:122] "RemoveContainer" containerID="2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3" Nov 02 21:01:51.880319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.880296 5136 scope.go:122] "RemoveContainer" containerID="2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3" Nov 02 21:01:51.880587 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:51.880568 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3\": container with ID starting with 2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3 not found: ID does not exist" containerID="2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3" Nov 02 21:01:51.880640 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:51.880594 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3"} err="failed to get container status \"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3\": rpc error: code = NotFound desc = could not find container \"2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3\": container with ID starting with 2d206dc4b6fec1ceba12d9832a89b0ebcfa3d3fc6076b4af3ef1d58908c26bb3 not found: ID does not exist" Nov 02 21:01:52.033134 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:52.033103 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 21:01:52.038089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:52.038061 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3109b-84795878fc-p2ldj"] Nov 02 21:01:52.547809 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:52.547777 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" path="/var/lib/kubelet/pods/9e1dfc8a-4822-4962-b96e-b4d02cad7581/volumes" Nov 02 21:01:56.167478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.167413 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 21:01:56.224587 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224555 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224600 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224637 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224699 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224816 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224845 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.224917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224857 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:56.224917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224887 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.225117 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224957 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:56.225117 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.224983 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.225117 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225026 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:56.225117 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225105 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\") pod \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\" (UID: \"1f8e0a95-34f4-4355-bfc5-258b1da17a5f\") " Nov 02 21:01:56.225343 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225130 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:56.225343 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225320 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:56.225668 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225646 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.225783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225672 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.225783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225687 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.225783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225700 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.225783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.225714 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.226959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.226936 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:01:56.227056 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.227044 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:56.227419 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.227398 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:56.227482 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.227402 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs" pod "1f8e0a95-34f4-4355-bfc5-258b1da17a5f" (UID: "1f8e0a95-34f4-4355-bfc5-258b1da17a5f"). InnerVolumeSpecName "kube-api-access-4mjhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:56.327560 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.327528 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.327560 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.327561 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.327872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.327574 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.327872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.327587 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjhs\" (UniqueName: \"kubernetes.io/projected/1f8e0a95-34f4-4355-bfc5-258b1da17a5f-kube-api-access-4mjhs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:56.902362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.902281 5136 generic.go:356] "Generic (PLEG): container finished" podID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerID="b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9" exitCode=0 Nov 02 21:01:56.902362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.902334 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" Nov 02 21:01:56.902362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.902337 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerDied","Data":"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9"} Nov 02 21:01:56.902682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.902392 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p" event={"ID":"1f8e0a95-34f4-4355-bfc5-258b1da17a5f","Type":"ContainerDied","Data":"b5b0fd6ef760cb788d3162f2a2a8acb496780b657818df027b3c854b3f540ff5"} Nov 02 21:01:56.902682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.902413 5136 scope.go:122] "RemoveContainer" containerID="b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9" Nov 02 21:01:56.912436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.912418 5136 scope.go:122] "RemoveContainer" containerID="ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850" Nov 02 21:01:56.921056 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.921038 5136 scope.go:122] "RemoveContainer" containerID="b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9" Nov 02 21:01:56.921317 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:56.921295 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9\": container with ID starting with b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9 not found: ID does not exist" containerID="b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9" Nov 02 21:01:56.921362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.921327 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9"} err="failed to get container status \"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9\": rpc error: code = NotFound desc = could not find container \"b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9\": container with ID starting with b25cdf04c63bbd01bba1e68c7bf0667d0e7e23221f5331c35851f2913aa87aa9 not found: ID does not exist" Nov 02 21:01:56.921362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.921344 5136 scope.go:122] "RemoveContainer" containerID="ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850" Nov 02 21:01:56.921595 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:01:56.921575 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850\": container with ID starting with ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850 not found: ID does not exist" containerID="ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850" Nov 02 21:01:56.921634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:56.921602 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850"} err="failed to get container status \"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850\": rpc error: code = NotFound desc = could not find container \"ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850\": container with ID starting with ebf06fd8859fc8208e552995c98a29b219cccf7996cb70b0afb76c9180fa8850 not found: ID does not exist" Nov 02 21:01:57.106914 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:57.106880 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 21:01:57.154779 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:57.154652 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f97409-openshift-default-c545ff9b9-xrt7p"] Nov 02 21:01:58.547339 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:01:58.547305 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" path="/var/lib/kubelet/pods/1f8e0a95-34f4-4355-bfc5-258b1da17a5f/volumes" Nov 02 21:02:10.679206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:10.679171 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 21:02:10.679618 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:10.679420 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" containerID="cri-o://ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862" gracePeriod=30 Nov 02 21:02:15.988426 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:15.988391 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 21:02:16.018871 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.018839 5136 generic.go:356] "Generic (PLEG): container finished" podID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerID="ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862" exitCode=0 Nov 02 21:02:16.019051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.018932 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" Nov 02 21:02:16.019051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.018963 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerDied","Data":"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862"} Nov 02 21:02:16.019051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.019026 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx" event={"ID":"44f9f9a2-ec05-4fcd-9e5e-8b010e801796","Type":"ContainerDied","Data":"fe490b8b8bf987488c7ac39d4a58811a2fbb7121d07d47fe23dacc3f78c06519"} Nov 02 21:02:16.019299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.019054 5136 scope.go:122] "RemoveContainer" containerID="ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862" Nov 02 21:02:16.029192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.029173 5136 scope.go:122] "RemoveContainer" containerID="058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a" Nov 02 21:02:16.042634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.042611 5136 scope.go:122] "RemoveContainer" containerID="ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862" Nov 02 21:02:16.043130 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:02:16.043096 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862\": container with ID starting with ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862 not found: ID does not exist" containerID="ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862" Nov 02 21:02:16.043286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.043141 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862"} err="failed to get container status \"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862\": rpc error: code = NotFound desc = could not find container \"ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862\": container with ID starting with ef5741d78cd689bbb52207dd75ee3e9d8a2599b9bf654270b5db0aa1d6c19862 not found: ID does not exist" Nov 02 21:02:16.043286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.043165 5136 scope.go:122] "RemoveContainer" containerID="058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a" Nov 02 21:02:16.043774 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:02:16.043745 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a\": container with ID starting with 058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a not found: ID does not exist" containerID="058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a" Nov 02 21:02:16.043872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.043793 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a"} err="failed to get container status \"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a\": rpc error: code = NotFound desc = could not find container \"058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a\": container with ID starting with 058e6a181b1e365a89742911e4be6973cd19aa49bf6dfa310558a07c8b4fbe1a not found: ID does not exist" Nov 02 21:02:16.085031 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.084998 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085174 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085100 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085174 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085169 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085200 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085237 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085314 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085304 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:16.085571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085352 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085438 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085449 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:16.085571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085472 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\") pod \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\" (UID: \"44f9f9a2-ec05-4fcd-9e5e-8b010e801796\") " Nov 02 21:02:16.085822 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085636 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:16.085950 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.085926 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:16.086169 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.086149 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:16.086330 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.086166 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.086413 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.086347 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.086413 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.086367 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.086413 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.086383 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.087699 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.087678 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "kube-api-access-bnn5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:16.087931 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.087913 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:02:16.088095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.088074 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:16.088095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.088078 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy" pod "44f9f9a2-ec05-4fcd-9e5e-8b010e801796" (UID: "44f9f9a2-ec05-4fcd-9e5e-8b010e801796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:16.187837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.187806 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.187837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.187830 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.187837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.187840 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnn5j\" (UniqueName: \"kubernetes.io/projected/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-kube-api-access-bnn5j\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.188076 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.187851 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.188076 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.187861 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f9f9a2-ec05-4fcd-9e5e-8b010e801796-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:16.438116 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.438028 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 21:02:16.443745 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.443707 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4530c5-openshift-default-7d9d5f6c9f-lc5lx"] Nov 02 21:02:16.547282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:02:16.547252 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" path="/var/lib/kubelet/pods/44f9f9a2-ec05-4fcd-9e5e-8b010e801796/volumes" Nov 02 21:02:21.004280 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:02:21.004249 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:02:51.005335 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:02:51.005297 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:03:21.005692 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:03:21.005649 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:03:33.067442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.067409 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:03:33.068008 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.067992 5136 cpu_manager.go:414] "containerMap: removing container" podUID="772c9378-13ae-44ec-93a1-24c271e49713" containerName="payload-pre-processing" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068008 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772c9378-13ae-44ec-93a1-24c271e49713" containerName="payload-pre-processing" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068028 5136 cpu_manager.go:414] "containerMap: removing container" podUID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" containerName="payload-processing" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068033 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" containerName="payload-processing" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068039 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068044 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068052 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ef68c87a-2391-4e67-a202-7804a6c1f380" containerName="payload-pre-processing" Nov 02 21:03:33.068063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068057 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef68c87a-2391-4e67-a202-7804a6c1f380" containerName="payload-pre-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068067 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" containerName="maas-api" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068072 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" containerName="maas-api" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068079 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068084 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068093 5136 cpu_manager.go:414] "containerMap: removing container" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068097 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068105 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068109 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068116 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068126 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068140 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068145 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068152 5136 cpu_manager.go:414] "containerMap: removing container" podUID="85b94f48-d953-458f-be3b-2505a67c7538" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068157 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b94f48-d953-458f-be3b-2505a67c7538" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068168 5136 cpu_manager.go:414] "containerMap: removing container" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068173 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068182 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068187 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068199 5136 cpu_manager.go:414] "containerMap: removing container" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068203 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068210 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068215 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068229 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068233 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82025e4-6b87-4eb2-a171-da812aa8e860" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068312 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4429f2-5ab9-4205-9ff6-5828958c1c54" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068319 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068329 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="772c9378-13ae-44ec-93a1-24c271e49713" containerName="payload-pre-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068336 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8a4baf-a24b-4cb5-a1d4-55c08df2a432" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068341 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068347 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef68c87a-2391-4e67-a202-7804a6c1f380" containerName="payload-pre-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068353 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1dfc8a-4822-4962-b96e-b4d02cad7581" containerName="maas-api" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068358 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66b6b27-5cb5-42f1-9709-ee27ac95695a" containerName="discovery" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068364 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068368 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068375 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4d711d-52b0-4e94-811a-6e70e2a6a633" containerName="cleanup" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068380 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a364c385-934d-4d00-ac4c-4c22ff3c4656" containerName="payload-processing" Nov 02 21:03:33.068372 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.068385 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b94f48-d953-458f-be3b-2505a67c7538" containerName="payload-processing" Nov 02 21:03:33.072045 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.072021 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.125122 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.125090 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:03:33.148084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.148053 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"openshift-service-ca.crt\"" Nov 02 21:03:33.169053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.169020 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"default-dockercfg-c8gsb\"" Nov 02 21:03:33.169053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.169020 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"kube-root-ca.crt\"" Nov 02 21:03:33.169243 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.169124 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"rate-test-model-76c687-kserve-self-signed-certs\"" Nov 02 21:03:33.176715 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.176690 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.176872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.176849 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.176926 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.176910 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.176972 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.176959 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.177070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.177059 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.177169 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.177148 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8gh\" (UniqueName: \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278404 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8gh\" (UniqueName: \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278452 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278508 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278528 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278571 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.278652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.278638 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.279089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.279062 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.279206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.279099 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.279206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.279063 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.280907 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.280887 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.281090 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.281073 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.332500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.332424 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8gh\" (UniqueName: \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\") pod \"rate-test-model-76c687-kserve-7c95cfd957-bqd9j\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.384830 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.384793 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:33.752899 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:33.752871 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:03:33.754426 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:03:33.754400 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22fdb42e_0ce5_4f59_9921_09a021a30af5.slice/crio-21c5881cd77c3e077e206c2f911a8174514bb69b44d7f1835c9647f787a18b61 WatchSource:0}: Error finding container 21c5881cd77c3e077e206c2f911a8174514bb69b44d7f1835c9647f787a18b61: Status 404 returned error can't find the container with id 21c5881cd77c3e077e206c2f911a8174514bb69b44d7f1835c9647f787a18b61 Nov 02 21:03:34.565544 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:34.565504 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" event={"ID":"22fdb42e-0ce5-4f59-9921-09a021a30af5","Type":"ContainerStarted","Data":"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b"} Nov 02 21:03:34.565980 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:34.565555 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" event={"ID":"22fdb42e-0ce5-4f59-9921-09a021a30af5","Type":"ContainerStarted","Data":"21c5881cd77c3e077e206c2f911a8174514bb69b44d7f1835c9647f787a18b61"} Nov 02 21:03:34.676060 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:34.676010 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" podStartSLOduration=1.67599672 podStartE2EDuration="1.67599672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:34.67304315 +0000 UTC m=+4498.279587648" watchObservedRunningTime="2026-11-02 21:03:34.67599672 +0000 UTC m=+4498.282541246" Nov 02 21:03:35.385297 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:35.385267 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:43.386014 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:43.385969 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:43.398421 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:43.398394 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:43.629370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:03:43.629342 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:03:51.007121 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:03:51.007033 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:04:21.004174 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:04:21.004139 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:04:41.530132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.530099 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:04:41.530809 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.530793 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f9f9a2-ec05-4fcd-9e5e-8b010e801796" containerName="istio-proxy" Nov 02 21:04:41.530865 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.530816 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8e0a95-34f4-4355-bfc5-258b1da17a5f" containerName="istio-proxy" Nov 02 21:04:41.534165 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.534148 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.632373 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.632337 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"openshift-service-ca.crt\"" Nov 02 21:04:41.632373 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.632355 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"kube-root-ca.crt\"" Nov 02 21:04:41.632581 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.632448 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"sub-test-model-a9ed60-kserve-self-signed-certs\"" Nov 02 21:04:41.632581 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.632529 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"default-dockercfg-bjw6w\"" Nov 02 21:04:41.635211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.635190 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:04:41.703979 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.703941 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.704210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.704052 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.704210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.704130 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.704366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.704215 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.705527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.705496 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.707043 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.705667 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx84n\" (UniqueName: \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807067 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.806979 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807067 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807040 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807070 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807164 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx84n\" (UniqueName: \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807199 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807265 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807542 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807613 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.807677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.807629 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.809333 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.809310 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.809674 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.809657 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:41.860957 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:41.860930 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx84n\" (UniqueName: \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\") pod \"sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:42.147076 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:42.146996 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:42.516188 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:42.516159 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:04:42.517692 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:04:42.517666 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaa2139e_b3c4_4f3d_8cb6_62437d5203fb.slice/crio-0329769e8a216c6136b3c1abfa671a91c1a4142061f0980a71fbf554aea72ec1 WatchSource:0}: Error finding container 0329769e8a216c6136b3c1abfa671a91c1a4142061f0980a71fbf554aea72ec1: Status 404 returned error can't find the container with id 0329769e8a216c6136b3c1abfa671a91c1a4142061f0980a71fbf554aea72ec1 Nov 02 21:04:42.987464 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:42.987428 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" event={"ID":"baa2139e-b3c4-4f3d-8cb6-62437d5203fb","Type":"ContainerStarted","Data":"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc"} Nov 02 21:04:42.987876 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:42.987473 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" event={"ID":"baa2139e-b3c4-4f3d-8cb6-62437d5203fb","Type":"ContainerStarted","Data":"0329769e8a216c6136b3c1abfa671a91c1a4142061f0980a71fbf554aea72ec1"} Nov 02 21:04:43.098922 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:43.098870 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" podStartSLOduration=2.098855764 podStartE2EDuration="2.098855764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:04:43.096769467 +0000 UTC m=+4566.703313965" watchObservedRunningTime="2026-11-02 21:04:43.098855764 +0000 UTC m=+4566.705400294" Nov 02 21:04:43.147640 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:43.147609 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:51.003915 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:04:51.003875 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:04:52.150147 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:52.149146 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:52.175355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:52.175316 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:53.075716 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:53.075678 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:04:53.447285 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:53.446900 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:04:53.447285 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:53.446981 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:04:53.500842 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:53.500806 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:04:53.501097 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:04:53.501070 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:05:21.005983 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:05:21.005904 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:05:39.845609 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.845573 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:05:39.850472 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.850451 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.931578 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.931553 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-68a6c923-openshift-default-dockercfg-wpppd\"" Nov 02 21:05:39.947315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947281 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947503 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947332 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947503 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947470 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947525 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947587 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947768 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947658 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947768 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947705 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947768 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947745 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7rm\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.947945 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.947786 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:39.953751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:39.953705 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050455 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050517 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050609 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050664 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050723 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050826 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050876 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050913 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7rm\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.050962 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.051762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.051429 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.052488 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.052449 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.052750 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.052709 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.054786 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.053715 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.054786 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.054277 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.062223 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.062196 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.064554 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.064526 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.105937 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.105872 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.107654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.107629 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7rm\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\") pod \"e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.164210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.164180 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:40.581022 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.580997 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:05:40.581629 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:05:40.581590 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73d6c30_028d_47cc_b37f_40227bda53f0.slice/crio-c42436c480d91d9499ca34860c79d516ddffd367347a39c48087dd27d66ca872 WatchSource:0}: Error finding container c42436c480d91d9499ca34860c79d516ddffd367347a39c48087dd27d66ca872: Status 404 returned error can't find the container with id c42436c480d91d9499ca34860c79d516ddffd367347a39c48087dd27d66ca872 Nov 02 21:05:40.584198 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.584165 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:05:40.584271 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.584227 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:05:40.584271 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:40.584257 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:05:41.368406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:41.368370 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerStarted","Data":"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a"} Nov 02 21:05:41.368833 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:41.368452 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerStarted","Data":"c42436c480d91d9499ca34860c79d516ddffd367347a39c48087dd27d66ca872"} Nov 02 21:05:41.480455 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:41.480403 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podStartSLOduration=2.480389994 podStartE2EDuration="2.480389994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:41.478267223 +0000 UTC m=+4625.084811735" watchObservedRunningTime="2026-11-02 21:05:41.480389994 +0000 UTC m=+4625.086934491" Nov 02 21:05:42.164338 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:42.164308 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:42.170483 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:42.170458 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:42.374534 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:42.374500 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:42.375407 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:42.375391 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:05:45.283690 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.283654 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:05:45.288047 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.288027 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305349 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305546 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305413 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305646 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305625 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqfm\" (UniqueName: \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305711 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305683 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305782 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305754 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.305851 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.305839 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.341153 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.341121 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:05:45.385120 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.385091 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"openshift-service-ca.crt\"" Nov 02 21:05:45.385290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.385186 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"default-dockercfg-kxl45\"" Nov 02 21:05:45.385290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.385186 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"kube-root-ca.crt\"" Nov 02 21:05:45.386550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.386536 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"rate-test-model-6b12ae-kserve-self-signed-certs\"" Nov 02 21:05:45.407371 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407338 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqfm\" (UniqueName: \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407525 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407386 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407525 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407405 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407525 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407460 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407525 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407517 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407776 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407548 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407990 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407971 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.407990 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.407988 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.408138 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.408020 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.409851 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.409832 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.410119 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.410102 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.462069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.462036 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqfm\" (UniqueName: \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\") pod \"rate-test-model-6b12ae-kserve-75c79657d5-5rvfn\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.610545 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.610472 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:45.982501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:45.982471 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:05:45.982939 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:05:45.982916 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb73808_afeb_4621_860c_8b7472ee389b.slice/crio-8009636646a7ceaf3dcd828e5d907083a6fe6617e1751489780669eb8c4a4d01 WatchSource:0}: Error finding container 8009636646a7ceaf3dcd828e5d907083a6fe6617e1751489780669eb8c4a4d01: Status 404 returned error can't find the container with id 8009636646a7ceaf3dcd828e5d907083a6fe6617e1751489780669eb8c4a4d01 Nov 02 21:05:46.401127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:46.401034 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" event={"ID":"ddb73808-afeb-4621-860c-8b7472ee389b","Type":"ContainerStarted","Data":"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251"} Nov 02 21:05:46.401127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:46.401085 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" event={"ID":"ddb73808-afeb-4621-860c-8b7472ee389b","Type":"ContainerStarted","Data":"8009636646a7ceaf3dcd828e5d907083a6fe6617e1751489780669eb8c4a4d01"} Nov 02 21:05:46.510462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:46.510413 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" podStartSLOduration=1.510400636 podStartE2EDuration="1.510400636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:46.508372248 +0000 UTC m=+4630.114916746" watchObservedRunningTime="2026-11-02 21:05:46.510400636 +0000 UTC m=+4630.116945178" Nov 02 21:05:46.611426 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:46.611396 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:51.031442 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:05:51.031395 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:05:51.276296 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.276256 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:05:51.280644 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.280621 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.336874 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.336841 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:05:51.380541 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.380508 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-68a6c923\"" Nov 02 21:05:51.380991 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.380973 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-68a6c923-dockercfg-w5h5g\"" Nov 02 21:05:51.397783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.390513 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.397783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.390619 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtmc\" (UniqueName: \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.491822 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.491791 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.492003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.491857 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtmc\" (UniqueName: \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.492501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.492481 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.546352 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.546275 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtmc\" (UniqueName: \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\") pod \"payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.595789 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.595761 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:51.964419 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:51.964391 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:05:51.966059 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:05:51.966028 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3bd798_e7ec_4b02_98bb_d4572a725141.slice/crio-ed5cf20ebb2f79b025a80fe051fd57dd44cdc66abf41d4f50941c695592e596c WatchSource:0}: Error finding container ed5cf20ebb2f79b025a80fe051fd57dd44cdc66abf41d4f50941c695592e596c: Status 404 returned error can't find the container with id ed5cf20ebb2f79b025a80fe051fd57dd44cdc66abf41d4f50941c695592e596c Nov 02 21:05:52.451860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:52.451827 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" event={"ID":"9c3bd798-e7ec-4b02-98bb-d4572a725141","Type":"ContainerStarted","Data":"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8"} Nov 02 21:05:52.452295 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:52.451876 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" event={"ID":"9c3bd798-e7ec-4b02-98bb-d4572a725141","Type":"ContainerStarted","Data":"ed5cf20ebb2f79b025a80fe051fd57dd44cdc66abf41d4f50941c695592e596c"} Nov 02 21:05:52.452295 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:52.451941 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:05:52.559458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:52.559409 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" podStartSLOduration=1.5593953809999999 podStartE2EDuration="1.559395381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:52.559286387 +0000 UTC m=+4636.165830886" watchObservedRunningTime="2026-11-02 21:05:52.559395381 +0000 UTC m=+4636.165939879" Nov 02 21:05:55.611635 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:55.611595 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:55.624900 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:55.624877 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:56.487222 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:56.487191 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:05:58.668739 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.668709 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:05:58.673388 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.673365 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.726709 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.726674 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:05:58.770437 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.770404 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"sub-test-model-578636-kserve-self-signed-certs\"" Nov 02 21:05:58.770848 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.770828 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"default-dockercfg-66vpb\"" Nov 02 21:05:58.770968 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.770852 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"openshift-service-ca.crt\"" Nov 02 21:05:58.770968 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.770943 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-578636\"/\"kube-root-ca.crt\"" Nov 02 21:05:58.784590 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.784564 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.784693 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.784605 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.784807 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.784790 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.784881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.784855 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9t6x\" (UniqueName: \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.784968 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.784954 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.785029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.785005 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886249 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886506 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886304 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886506 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886428 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886506 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886490 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9t6x\" (UniqueName: \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886687 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886562 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886687 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886610 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.886815 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.886775 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.887054 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.887029 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.887135 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.887099 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.889018 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.888988 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.889604 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.889579 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.941199 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.941133 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9t6x\" (UniqueName: \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\") pod \"sub-test-model-578636-kserve-67895489f9-jp5b2\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:58.986348 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:58.986317 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:05:59.404541 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:59.404512 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:05:59.405598 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:05:59.405567 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2796ae70_7b8c_43d1_8122_e06c738fd254.slice/crio-a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419 WatchSource:0}: Error finding container a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419: Status 404 returned error can't find the container with id a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419 Nov 02 21:05:59.498542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:59.498507 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" event={"ID":"2796ae70-7b8c-43d1-8122-e06c738fd254","Type":"ContainerStarted","Data":"d999752d889c36295a46044db8f810aea01f2027f178bd071e86d1c68557db71"} Nov 02 21:05:59.498704 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:59.498560 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" event={"ID":"2796ae70-7b8c-43d1-8122-e06c738fd254","Type":"ContainerStarted","Data":"a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419"} Nov 02 21:05:59.607861 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:59.607811 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" podStartSLOduration=1.607797119 podStartE2EDuration="1.607797119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:59.606926485 +0000 UTC m=+4643.213470983" watchObservedRunningTime="2026-11-02 21:05:59.607797119 +0000 UTC m=+4643.214341616" Nov 02 21:05:59.987040 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:05:59.987014 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:06:03.462171 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:03.462136 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:06:08.987253 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:08.987218 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:06:09.000190 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:09.000164 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:06:09.573049 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:09.573018 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:06:21.005962 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:06:21.005927 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:06:51.004809 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:06:51.004714 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:06:53.056737 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.056691 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:06:53.060944 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.060913 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.110334 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.110308 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-68a6c923-serving-cert\"" Nov 02 21:06:53.152915 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.152887 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:06:53.176158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.176109 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.176365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.176172 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.176365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.176287 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5db9\" (UniqueName: \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.176501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.176390 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.277370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.277333 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.277562 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.277383 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.277562 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.277415 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5db9\" (UniqueName: \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.277562 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.277462 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.277781 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:06:53.277578 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-serving-cert: secret "maas-api-e2e-mt-68a6c923-serving-cert" not found Nov 02 21:06:53.277781 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:06:53.277659 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls podName:965a253c-ede8-4374-b612-491c18c0461c nodeName:}" failed. No retries permitted until 2026-11-02 21:06:53.777636027 +0000 UTC m=+4697.384180520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls") pod "maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" (UID: "965a253c-ede8-4374-b612-491c18c0461c") : secret "maas-api-e2e-mt-68a6c923-serving-cert" not found Nov 02 21:06:53.278345 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.278325 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.279925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.279906 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.331946 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.331870 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5db9\" (UniqueName: \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.784391 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.784354 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.786949 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.786910 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") pod \"maas-api-e2e-mt-68a6c923-5c978f8896-76qhf\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:53.975390 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:53.975359 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:54.343092 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:54.343066 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:06:54.344466 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:06:54.344440 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965a253c_ede8_4374_b612_491c18c0461c.slice/crio-557a81d6936a27c2850cab6eda4e0b3d45de5641739fc21bfd46db7c7881c249 WatchSource:0}: Error finding container 557a81d6936a27c2850cab6eda4e0b3d45de5641739fc21bfd46db7c7881c249: Status 404 returned error can't find the container with id 557a81d6936a27c2850cab6eda4e0b3d45de5641739fc21bfd46db7c7881c249 Nov 02 21:06:54.345995 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:54.345975 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:06:54.846223 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:54.846180 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" event={"ID":"965a253c-ede8-4374-b612-491c18c0461c","Type":"ContainerStarted","Data":"557a81d6936a27c2850cab6eda4e0b3d45de5641739fc21bfd46db7c7881c249"} Nov 02 21:06:55.858462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:55.858386 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" event={"ID":"965a253c-ede8-4374-b612-491c18c0461c","Type":"ContainerStarted","Data":"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac"} Nov 02 21:06:55.858901 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:55.858520 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:06:55.969985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:06:55.969934 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" podStartSLOduration=1.719101465 podStartE2EDuration="2.969918377s" totalImagesPullingTime="1.250816912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:06:55.967989811 +0000 UTC m=+4699.574534314" watchObservedRunningTime="2026-11-02 21:06:55.969918377 +0000 UTC m=+4699.576462904" Nov 02 21:07:01.869857 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:01.869826 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:07:21.025174 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:07:21.025140 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:07:28.401376 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:28.401325 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:07:28.401823 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:28.401412 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:07:29.080035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.080000 5136 generic.go:356] "Generic (PLEG): container finished" podID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerID="d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a" exitCode=0 Nov 02 21:07:29.080236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.080073 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerDied","Data":"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a"} Nov 02 21:07:29.080477 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.080461 5136 scope.go:122] "RemoveContainer" containerID="d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a" Nov 02 21:07:29.081969 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.081937 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:07:29.082068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.082007 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:07:29.082068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:29.082041 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:07:30.113607 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:30.113556 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerStarted","Data":"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce"} Nov 02 21:07:30.164692 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:30.164665 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:07:30.164943 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:30.164922 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 21:07:30.164994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:30.164977 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 21:07:31.165532 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:31.165495 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 21:07:31.166065 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:31.165566 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 21:07:32.164943 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:32.164906 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 21:07:32.165137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:32.164963 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 21:07:33.165001 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:33.164970 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 21:07:33.165395 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:33.165026 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 21:07:34.165358 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:34.165322 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 21:07:34.165837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:34.165376 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 21:07:35.201213 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:35.201180 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:07:35.202029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:35.202004 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:07:35.202951 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:07:35.202926 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:07:51.004005 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:07:51.003972 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:08:15.608136 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:15.608096 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 21:08:20.608978 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:20.608869 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" containerID="cri-o://3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd" gracePeriod=25 Nov 02 21:08:21.004695 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:08:21.004657 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:08:25.436037 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:25.435996 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:08:35.435488 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:35.435452 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:08:36.002601 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.002569 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 21:08:36.002819 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.002792 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" podUID="6e94ace9-883e-4886-82e6-b8547bcdee50" containerName="maas-api" containerID="cri-o://ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a" gracePeriod=30 Nov 02 21:08:36.290456 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.290427 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 21:08:36.431231 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.431200 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\" (UniqueName: \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\") pod \"6e94ace9-883e-4886-82e6-b8547bcdee50\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " Nov 02 21:08:36.431422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.431277 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\") pod \"6e94ace9-883e-4886-82e6-b8547bcdee50\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " Nov 02 21:08:36.431422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.431305 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") pod \"6e94ace9-883e-4886-82e6-b8547bcdee50\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " Nov 02 21:08:36.431422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.431369 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\") pod \"6e94ace9-883e-4886-82e6-b8547bcdee50\" (UID: \"6e94ace9-883e-4886-82e6-b8547bcdee50\") " Nov 02 21:08:36.431916 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.431882 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca" pod "6e94ace9-883e-4886-82e6-b8547bcdee50" (UID: "6e94ace9-883e-4886-82e6-b8547bcdee50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:08:36.432176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.432152 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e94ace9-883e-4886-82e6-b8547bcdee50-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:36.433912 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.433877 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc" pod "6e94ace9-883e-4886-82e6-b8547bcdee50" (UID: "6e94ace9-883e-4886-82e6-b8547bcdee50"). InnerVolumeSpecName "kube-api-access-j97pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:08:36.434181 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.434144 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs" pod "6e94ace9-883e-4886-82e6-b8547bcdee50" (UID: "6e94ace9-883e-4886-82e6-b8547bcdee50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:08:36.434369 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.434349 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls" pod "6e94ace9-883e-4886-82e6-b8547bcdee50" (UID: "6e94ace9-883e-4886-82e6-b8547bcdee50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:08:36.533805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.533680 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97pc\" (UniqueName: \"kubernetes.io/projected/6e94ace9-883e-4886-82e6-b8547bcdee50-kube-api-access-j97pc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:36.533805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.533715 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:36.533805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.533744 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e94ace9-883e-4886-82e6-b8547bcdee50-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:36.546200 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.546149 5136 generic.go:356] "Generic (PLEG): container finished" podID="6e94ace9-883e-4886-82e6-b8547bcdee50" containerID="ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a" exitCode=0 Nov 02 21:08:36.552361 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.552317 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" Nov 02 21:08:36.560609 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.560580 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" event={"ID":"6e94ace9-883e-4886-82e6-b8547bcdee50","Type":"ContainerDied","Data":"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a"} Nov 02 21:08:36.560849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.560637 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn" event={"ID":"6e94ace9-883e-4886-82e6-b8547bcdee50","Type":"ContainerDied","Data":"caafcf166c6c59eb581e6fc0e4be51e0bf73ce02be9731690b02adb8abaf0fff"} Nov 02 21:08:36.560849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.560674 5136 scope.go:122] "RemoveContainer" containerID="ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a" Nov 02 21:08:36.574607 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.574588 5136 scope.go:122] "RemoveContainer" containerID="ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a" Nov 02 21:08:36.574978 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:08:36.574957 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a\": container with ID starting with ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a not found: ID does not exist" containerID="ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a" Nov 02 21:08:36.575092 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.574987 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a"} err="failed to get container status \"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a\": rpc error: code = NotFound desc = could not find container \"ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a\": container with ID starting with ae2b171e59b8de2dd63830735a998f6054a3c219aca2a66decfdb96df63f530a not found: ID does not exist" Nov 02 21:08:36.709904 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.709875 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 21:08:36.714908 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:36.714879 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2c6ce9-5d58f64444-5lnnn"] Nov 02 21:08:38.548367 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:38.548336 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e94ace9-883e-4886-82e6-b8547bcdee50" path="/var/lib/kubelet/pods/6e94ace9-883e-4886-82e6-b8547bcdee50/volumes" Nov 02 21:08:40.886419 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:40.886392 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 21:08:40.988573 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:40.988541 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\") pod \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " Nov 02 21:08:40.988777 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:40.988619 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\" (UniqueName: \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\") pod \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\" (UID: \"8f98289b-28b8-4294-8e9c-2ea6b459c7cb\") " Nov 02 21:08:40.988938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:40.988918 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume" pod "8f98289b-28b8-4294-8e9c-2ea6b459c7cb" (UID: "8f98289b-28b8-4294-8e9c-2ea6b459c7cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:08:40.990805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:40.990782 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz" pod "8f98289b-28b8-4294-8e9c-2ea6b459c7cb" (UID: "8f98289b-28b8-4294-8e9c-2ea6b459c7cb"). InnerVolumeSpecName "kube-api-access-wdzwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:08:41.090239 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.090213 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:41.090239 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.090237 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdzwz\" (UniqueName: \"kubernetes.io/projected/8f98289b-28b8-4294-8e9c-2ea6b459c7cb-kube-api-access-wdzwz\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:41.579893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.579858 5136 generic.go:356] "Generic (PLEG): container finished" podID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerID="3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd" exitCode=0 Nov 02 21:08:41.580086 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.579909 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" Nov 02 21:08:41.580086 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.579946 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" event={"ID":"8f98289b-28b8-4294-8e9c-2ea6b459c7cb","Type":"ContainerDied","Data":"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd"} Nov 02 21:08:41.580086 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.580003 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk" event={"ID":"8f98289b-28b8-4294-8e9c-2ea6b459c7cb","Type":"ContainerDied","Data":"5e367fdfc261fc935e1fe6baef8c7f03c32ee6b1508f9bd8eed67cddd8c6a87b"} Nov 02 21:08:41.580086 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.580029 5136 scope.go:122] "RemoveContainer" containerID="3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd" Nov 02 21:08:41.589971 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.589952 5136 scope.go:122] "RemoveContainer" containerID="3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd" Nov 02 21:08:41.590238 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:08:41.590209 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd\": container with ID starting with 3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd not found: ID does not exist" containerID="3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd" Nov 02 21:08:41.590303 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.590239 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd"} err="failed to get container status \"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd\": rpc error: code = NotFound desc = could not find container \"3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd\": container with ID starting with 3698f9f328d38381daa2068c19b37fcfbbec19379008617b24c44be1655333fd not found: ID does not exist" Nov 02 21:08:41.736221 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.736189 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 21:08:41.783437 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:41.783394 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2c6ce9-7688785949-xv6mk"] Nov 02 21:08:42.547554 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:08:42.547520 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" path="/var/lib/kubelet/pods/8f98289b-28b8-4294-8e9c-2ea6b459c7cb/volumes" Nov 02 21:08:51.003829 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:08:51.003792 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:09:21.004651 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:09:21.004609 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:09:51.005504 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:09:51.005419 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:09:57.034850 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:09:57.034807 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:10:02.035350 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.035306 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" podUID="9c3bd798-e7ec-4b02-98bb-d4572a725141" containerName="payload-pre-processing" containerID="cri-o://6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8" gracePeriod=25 Nov 02 21:10:02.297626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.297570 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:10:02.422744 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.422704 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7522-2a044d"] Nov 02 21:10:02.423354 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423335 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" Nov 02 21:10:02.423400 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423357 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" Nov 02 21:10:02.423400 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423396 5136 cpu_manager.go:414] "containerMap: removing container" podUID="6e94ace9-883e-4886-82e6-b8547bcdee50" containerName="maas-api" Nov 02 21:10:02.423468 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423402 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e94ace9-883e-4886-82e6-b8547bcdee50" containerName="maas-api" Nov 02 21:10:02.423468 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423416 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9c3bd798-e7ec-4b02-98bb-d4572a725141" containerName="payload-pre-processing" Nov 02 21:10:02.423468 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423421 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3bd798-e7ec-4b02-98bb-d4572a725141" containerName="payload-pre-processing" Nov 02 21:10:02.423576 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423519 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e94ace9-883e-4886-82e6-b8547bcdee50" containerName="maas-api" Nov 02 21:10:02.423576 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423529 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3bd798-e7ec-4b02-98bb-d4572a725141" containerName="payload-pre-processing" Nov 02 21:10:02.423576 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.423541 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f98289b-28b8-4294-8e9c-2ea6b459c7cb" containerName="payload-processing" Nov 02 21:10:02.426995 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.426978 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:02.458964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.458938 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\") pod \"9c3bd798-e7ec-4b02-98bb-d4572a725141\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " Nov 02 21:10:02.459185 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.459168 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\" (UniqueName: \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\") pod \"9c3bd798-e7ec-4b02-98bb-d4572a725141\" (UID: \"9c3bd798-e7ec-4b02-98bb-d4572a725141\") " Nov 02 21:10:02.459336 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.459313 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume" pod "9c3bd798-e7ec-4b02-98bb-d4572a725141" (UID: "9c3bd798-e7ec-4b02-98bb-d4572a725141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:02.459575 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.459556 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3bd798-e7ec-4b02-98bb-d4572a725141-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:02.461241 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.461209 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc" pod "9c3bd798-e7ec-4b02-98bb-d4572a725141" (UID: "9c3bd798-e7ec-4b02-98bb-d4572a725141"). InnerVolumeSpecName "kube-api-access-8jtmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:02.476621 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.476597 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bkzxn\"" Nov 02 21:10:02.477891 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.477715 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-2a044d"] Nov 02 21:10:02.560374 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.560347 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxml5\" (UniqueName: \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\") pod \"test-curl-7522-2a044d\" (UID: \"8b0682ea-d685-406e-aa9b-51e2cfb24e6b\") " pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:02.560582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.560563 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtmc\" (UniqueName: \"kubernetes.io/projected/9c3bd798-e7ec-4b02-98bb-d4572a725141-kube-api-access-8jtmc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:02.662068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.662033 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxml5\" (UniqueName: \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\") pod \"test-curl-7522-2a044d\" (UID: \"8b0682ea-d685-406e-aa9b-51e2cfb24e6b\") " pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:02.716009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.715974 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxml5\" (UniqueName: \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\") pod \"test-curl-7522-2a044d\" (UID: \"8b0682ea-d685-406e-aa9b-51e2cfb24e6b\") " pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:02.740505 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:02.740475 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:03.097102 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.097063 5136 generic.go:356] "Generic (PLEG): container finished" podID="9c3bd798-e7ec-4b02-98bb-d4572a725141" containerID="6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8" exitCode=0 Nov 02 21:10:03.097526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.097128 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" Nov 02 21:10:03.097526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.097140 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" event={"ID":"9c3bd798-e7ec-4b02-98bb-d4572a725141","Type":"ContainerDied","Data":"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8"} Nov 02 21:10:03.097526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.097198 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47" event={"ID":"9c3bd798-e7ec-4b02-98bb-d4572a725141","Type":"ContainerDied","Data":"ed5cf20ebb2f79b025a80fe051fd57dd44cdc66abf41d4f50941c695592e596c"} Nov 02 21:10:03.097526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.097223 5136 scope.go:122] "RemoveContainer" containerID="6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8" Nov 02 21:10:03.106964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.106937 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7522-2a044d"] Nov 02 21:10:03.108037 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.108016 5136 scope.go:122] "RemoveContainer" containerID="6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8" Nov 02 21:10:03.108317 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:03.108292 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8\": container with ID starting with 6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8 not found: ID does not exist" containerID="6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8" Nov 02 21:10:03.108380 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.108330 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8"} err="failed to get container status \"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8\": rpc error: code = NotFound desc = could not find container \"6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8\": container with ID starting with 6f4ac995bf3fca2700c88322666978b1cfc505cc5e48378205e729f0abdd4dd8 not found: ID does not exist" Nov 02 21:10:03.109922 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:10:03.109896 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b0682ea_d685_406e_aa9b_51e2cfb24e6b.slice/crio-36990102cecf5761125c42e7566ae5637de006efbdaf1a112f6106c79a3c18cb WatchSource:0}: Error finding container 36990102cecf5761125c42e7566ae5637de006efbdaf1a112f6106c79a3c18cb: Status 404 returned error can't find the container with id 36990102cecf5761125c42e7566ae5637de006efbdaf1a112f6106c79a3c18cb Nov 02 21:10:03.253170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.253138 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:10:03.257132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:03.257106 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68a6c923-b4cb858ff-lzn47"] Nov 02 21:10:04.109331 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:04.106986 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-2a044d" event={"ID":"8b0682ea-d685-406e-aa9b-51e2cfb24e6b","Type":"ContainerStarted","Data":"0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790"} Nov 02 21:10:04.109331 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:04.107037 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7522-2a044d" event={"ID":"8b0682ea-d685-406e-aa9b-51e2cfb24e6b","Type":"ContainerStarted","Data":"36990102cecf5761125c42e7566ae5637de006efbdaf1a112f6106c79a3c18cb"} Nov 02 21:10:04.217850 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:04.217799 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7522-2a044d" podStartSLOduration=2.217783508 podStartE2EDuration="2.217783508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:04.215990015 +0000 UTC m=+4887.822534513" watchObservedRunningTime="2026-11-02 21:10:04.217783508 +0000 UTC m=+4887.824327985" Nov 02 21:10:04.547776 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:04.547747 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3bd798-e7ec-4b02-98bb-d4572a725141" path="/var/lib/kubelet/pods/9c3bd798-e7ec-4b02-98bb-d4572a725141/volumes" Nov 02 21:10:05.705156 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:05.705123 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7522-2a044d"] Nov 02 21:10:05.712929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:05.712888 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7522-2a044d"] Nov 02 21:10:06.118453 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:06.118364 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7522-2a044d" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" containerName="test-curl-7522-2a044d" containerID="cri-o://0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790" gracePeriod=2 Nov 02 21:10:06.166968 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:06.166921 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-2a044d\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:06.591723 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:06.591687 5136 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7522-2a044d\" is forbidden: User \"system:node:ip-10-0-128-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-34.us-west-2.compute.internal' and this object" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:08.377304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:08.377243 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:08.441012 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:08.440981 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\" (UniqueName: \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\") pod \"8b0682ea-d685-406e-aa9b-51e2cfb24e6b\" (UID: \"8b0682ea-d685-406e-aa9b-51e2cfb24e6b\") " Nov 02 21:10:08.443279 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:08.443253 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5" pod "8b0682ea-d685-406e-aa9b-51e2cfb24e6b" (UID: "8b0682ea-d685-406e-aa9b-51e2cfb24e6b"). InnerVolumeSpecName "kube-api-access-mxml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:08.541972 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:08.541946 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxml5\" (UniqueName: \"kubernetes.io/projected/8b0682ea-d685-406e-aa9b-51e2cfb24e6b-kube-api-access-mxml5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:08.546865 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:08.546841 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" path="/var/lib/kubelet/pods/8b0682ea-d685-406e-aa9b-51e2cfb24e6b/volumes" Nov 02 21:10:09.138625 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:09.138588 5136 generic.go:356] "Generic (PLEG): container finished" podID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" containerID="0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790" exitCode=137 Nov 02 21:10:09.138826 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:09.138632 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7522-2a044d" Nov 02 21:10:09.138826 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:09.138687 5136 scope.go:122] "RemoveContainer" containerID="0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790" Nov 02 21:10:09.147838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:09.147817 5136 scope.go:122] "RemoveContainer" containerID="0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790" Nov 02 21:10:09.148106 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:09.148087 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790\": container with ID starting with 0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790 not found: ID does not exist" containerID="0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790" Nov 02 21:10:09.148170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:09.148120 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790"} err="failed to get container status \"0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790\": rpc error: code = NotFound desc = could not find container \"0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790\": container with ID starting with 0227f427ff315cf186814593185759ddd96c287e16c34ec1416caaf600964790 not found: ID does not exist" Nov 02 21:10:10.752084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:10.752047 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 21:10:19.426650 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.426615 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 21:10:19.968031 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.967998 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:10:19.968743 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.968714 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" containerName="test-curl-7522-2a044d" Nov 02 21:10:19.968824 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.968757 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" containerName="test-curl-7522-2a044d" Nov 02 21:10:19.968877 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.968867 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0682ea-d685-406e-aa9b-51e2cfb24e6b" containerName="test-curl-7522-2a044d" Nov 02 21:10:19.972451 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:19.972430 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.023043 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.023012 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-578636-openshift-default-dockercfg-gqmxj\"" Nov 02 21:10:20.026569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.026545 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:10:20.079216 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079185 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079301 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079347 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079379 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079409 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079546 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079526 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfn76\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079614 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079595 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079623 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.079724 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.079709 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.180802 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180764 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.180984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180841 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.180984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180874 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.180984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180904 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.180984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180961 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfn76\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181217 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.180992 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181217 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181014 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181217 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181093 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181217 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181196 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181328 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181193 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181398 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181573 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181482 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.181953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.181937 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.183219 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.183185 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.183496 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.183477 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.238908 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.238800 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.239311 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.239280 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfn76\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.286484 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.286382 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:20.720791 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.719688 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:10:20.745463 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:10:20.745427 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10baf3a1_6efd_4f99_8cd0_87a3a105666d.slice/crio-db9a40dcf73a2f101dbf704208dc84cddd5e406e85a15c24412ff17f5bce3b03 WatchSource:0}: Error finding container db9a40dcf73a2f101dbf704208dc84cddd5e406e85a15c24412ff17f5bce3b03: Status 404 returned error can't find the container with id db9a40dcf73a2f101dbf704208dc84cddd5e406e85a15c24412ff17f5bce3b03 Nov 02 21:10:20.748340 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.748300 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:10:20.748425 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.748395 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:10:20.748465 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:20.748438 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:10:21.008773 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:21.008670 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:10:21.219418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.219384 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" event={"ID":"10baf3a1-6efd-4f99-8cd0-87a3a105666d","Type":"ContainerStarted","Data":"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0"} Nov 02 21:10:21.219622 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.219434 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" event={"ID":"10baf3a1-6efd-4f99-8cd0-87a3a105666d","Type":"ContainerStarted","Data":"db9a40dcf73a2f101dbf704208dc84cddd5e406e85a15c24412ff17f5bce3b03"} Nov 02 21:10:21.287156 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.287064 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:21.288836 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.288809 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:21.288989 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.288855 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:21.390760 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:21.390240 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podStartSLOduration=2.390220752 podStartE2EDuration="2.390220752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:21.381409472 +0000 UTC m=+4904.987953961" watchObservedRunningTime="2026-11-02 21:10:21.390220752 +0000 UTC m=+4904.996765253" Nov 02 21:10:22.287436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:22.287401 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:22.287943 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:22.287456 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:22.670298 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:22.670218 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:10:22.670467 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:22.670404 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" podUID="965a253c-ede8-4374-b612-491c18c0461c" containerName="maas-api" containerID="cri-o://d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac" gracePeriod=30 Nov 02 21:10:23.044010 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.043979 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:10:23.142148 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142115 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\" (UniqueName: \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\") pod \"965a253c-ede8-4374-b612-491c18c0461c\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " Nov 02 21:10:23.142349 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142170 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") pod \"965a253c-ede8-4374-b612-491c18c0461c\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " Nov 02 21:10:23.142349 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142199 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\") pod \"965a253c-ede8-4374-b612-491c18c0461c\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " Nov 02 21:10:23.142349 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142235 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\") pod \"965a253c-ede8-4374-b612-491c18c0461c\" (UID: \"965a253c-ede8-4374-b612-491c18c0461c\") " Nov 02 21:10:23.142579 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142552 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca" pod "965a253c-ede8-4374-b612-491c18c0461c" (UID: "965a253c-ede8-4374-b612-491c18c0461c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:23.142848 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.142822 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a253c-ede8-4374-b612-491c18c0461c-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.144458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.144430 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls" pod "965a253c-ede8-4374-b612-491c18c0461c" (UID: "965a253c-ede8-4374-b612-491c18c0461c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:23.144458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.144446 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9" pod "965a253c-ede8-4374-b612-491c18c0461c" (UID: "965a253c-ede8-4374-b612-491c18c0461c"). InnerVolumeSpecName "kube-api-access-m5db9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:23.144458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.144459 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs" pod "965a253c-ede8-4374-b612-491c18c0461c" (UID: "965a253c-ede8-4374-b612-491c18c0461c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.244913 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5db9\" (UniqueName: \"kubernetes.io/projected/965a253c-ede8-4374-b612-491c18c0461c-kube-api-access-m5db9\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.244949 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.244964 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/965a253c-ede8-4374-b612-491c18c0461c-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.252572 5136 generic.go:356] "Generic (PLEG): container finished" podID="965a253c-ede8-4374-b612-491c18c0461c" containerID="d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac" exitCode=0 Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.253496 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.255163 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" event={"ID":"965a253c-ede8-4374-b612-491c18c0461c","Type":"ContainerDied","Data":"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac"} Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.255225 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf" event={"ID":"965a253c-ede8-4374-b612-491c18c0461c","Type":"ContainerDied","Data":"557a81d6936a27c2850cab6eda4e0b3d45de5641739fc21bfd46db7c7881c249"} Nov 02 21:10:23.256783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.255257 5136 scope.go:122] "RemoveContainer" containerID="d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac" Nov 02 21:10:23.281793 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.281763 5136 scope.go:122] "RemoveContainer" containerID="d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac" Nov 02 21:10:23.282415 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:23.282390 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac\": container with ID starting with d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac not found: ID does not exist" containerID="d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac" Nov 02 21:10:23.282499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.282427 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac"} err="failed to get container status \"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac\": rpc error: code = NotFound desc = could not find container \"d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac\": container with ID starting with d5fc7915d3f7776216857040aa69591a4b6b427107c736393fb001a69ef70eac not found: ID does not exist" Nov 02 21:10:23.286831 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.286806 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:23.287139 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.287117 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:23.420279 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.420243 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:10:23.423608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:23.423580 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68a6c923-5c978f8896-76qhf"] Nov 02 21:10:24.287620 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:24.287586 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:24.287878 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:24.287640 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:24.427254 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:24.427213 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" containerID="cri-o://23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572" gracePeriod=25 Nov 02 21:10:24.546978 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:24.546906 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965a253c-ede8-4374-b612-491c18c0461c" path="/var/lib/kubelet/pods/965a253c-ede8-4374-b612-491c18c0461c/volumes" Nov 02 21:10:25.286951 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:25.286914 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:25.287150 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:25.286968 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:25.774563 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:25.774519 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" containerID="cri-o://1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c" gracePeriod=45 Nov 02 21:10:26.122403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.122375 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 21:10:26.181489 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181450 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.181689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181510 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.182006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181688 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.182006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181799 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.182006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181803 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.182006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181845 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\" (UniqueName: \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.182006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.181928 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\") pod \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\" (UID: \"b4053b5f-e54a-405a-bbf9-bd67a1650d4a\") " Nov 02 21:10:26.182483 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.182459 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.182626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.182515 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.182626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.182578 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.182626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.182600 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.184236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.184211 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:26.184626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.184599 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.184626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.184601 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82" pod "b4053b5f-e54a-405a-bbf9-bd67a1650d4a" (UID: "b4053b5f-e54a-405a-bbf9-bd67a1650d4a"). InnerVolumeSpecName "kube-api-access-8ps82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:26.277887 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.277851 5136 generic.go:356] "Generic (PLEG): container finished" podID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerID="1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c" exitCode=0 Nov 02 21:10:26.278070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.277897 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" Nov 02 21:10:26.278070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.277918 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" event={"ID":"b4053b5f-e54a-405a-bbf9-bd67a1650d4a","Type":"ContainerDied","Data":"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c"} Nov 02 21:10:26.278070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.277952 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58" event={"ID":"b4053b5f-e54a-405a-bbf9-bd67a1650d4a","Type":"ContainerDied","Data":"38f47d451a1f25f92942b07d776f885c9f7950c18a29c8c268ce5f1862ea9261"} Nov 02 21:10:26.278070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.277974 5136 scope.go:122] "RemoveContainer" containerID="1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c" Nov 02 21:10:26.284394 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.284365 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.284505 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.284396 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.284505 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.284411 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.284505 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.284424 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ps82\" (UniqueName: \"kubernetes.io/projected/b4053b5f-e54a-405a-bbf9-bd67a1650d4a-kube-api-access-8ps82\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.287145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.287002 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 21:10:26.287145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.287069 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 21:10:26.313969 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.308897 5136 scope.go:122] "RemoveContainer" containerID="1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c" Nov 02 21:10:26.313969 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:26.309977 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c\": container with ID starting with 1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c not found: ID does not exist" containerID="1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c" Nov 02 21:10:26.313969 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.310014 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c"} err="failed to get container status \"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c\": rpc error: code = NotFound desc = could not find container \"1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c\": container with ID starting with 1ee009af3b73b6620f4788ca31a22779fc60ee1a78fe25a718ff391db055797c not found: ID does not exist" Nov 02 21:10:26.434280 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.434200 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 21:10:26.436787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.436760 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-2-w2-426941-kserve-68b8c94948-bsr58"] Nov 02 21:10:26.547571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:26.547538 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" path="/var/lib/kubelet/pods/b4053b5f-e54a-405a-bbf9-bd67a1650d4a/volumes" Nov 02 21:10:27.429074 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:27.428957 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:27.429660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:27.429547 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:27.430477 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:27.430455 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:10:31.814353 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:31.814310 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:10:34.156278 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.156241 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 21:10:34.156706 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.156439 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" podUID="d072c3af-98c3-4a14-9810-c1385806e003" containerName="maas-api" containerID="cri-o://c1b81c27b1e9c64355b38fbe3682c7c573c3cb0d3a2aec59dd49fe44ee10576e" gracePeriod=30 Nov 02 21:10:34.340960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.340844 5136 generic.go:356] "Generic (PLEG): container finished" podID="d072c3af-98c3-4a14-9810-c1385806e003" containerID="c1b81c27b1e9c64355b38fbe3682c7c573c3cb0d3a2aec59dd49fe44ee10576e" exitCode=0 Nov 02 21:10:34.340960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.340921 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" event={"ID":"d072c3af-98c3-4a14-9810-c1385806e003","Type":"ContainerDied","Data":"c1b81c27b1e9c64355b38fbe3682c7c573c3cb0d3a2aec59dd49fe44ee10576e"} Nov 02 21:10:34.456160 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.456129 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 21:10:34.605418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.605381 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\" (UniqueName: \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\") pod \"d072c3af-98c3-4a14-9810-c1385806e003\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " Nov 02 21:10:34.605599 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.605449 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") pod \"d072c3af-98c3-4a14-9810-c1385806e003\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " Nov 02 21:10:34.605661 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.605649 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\") pod \"d072c3af-98c3-4a14-9810-c1385806e003\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " Nov 02 21:10:34.605701 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.605678 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\") pod \"d072c3af-98c3-4a14-9810-c1385806e003\" (UID: \"d072c3af-98c3-4a14-9810-c1385806e003\") " Nov 02 21:10:34.606162 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.606131 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca" pod "d072c3af-98c3-4a14-9810-c1385806e003" (UID: "d072c3af-98c3-4a14-9810-c1385806e003"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:34.607751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.607711 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw" pod "d072c3af-98c3-4a14-9810-c1385806e003" (UID: "d072c3af-98c3-4a14-9810-c1385806e003"). InnerVolumeSpecName "kube-api-access-gwtgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:34.607853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.607791 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls" pod "d072c3af-98c3-4a14-9810-c1385806e003" (UID: "d072c3af-98c3-4a14-9810-c1385806e003"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:34.607853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.607845 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs" pod "d072c3af-98c3-4a14-9810-c1385806e003" (UID: "d072c3af-98c3-4a14-9810-c1385806e003"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:34.707676 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.707645 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:34.707676 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.707674 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072c3af-98c3-4a14-9810-c1385806e003-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:34.707967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.707685 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtgw\" (UniqueName: \"kubernetes.io/projected/d072c3af-98c3-4a14-9810-c1385806e003-kube-api-access-gwtgw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:34.707967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.707696 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072c3af-98c3-4a14-9810-c1385806e003-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:34.957547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.957513 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:10:34.957889 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:34.957806 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" containerID="cri-o://459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce" gracePeriod=30 Nov 02 21:10:35.039181 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.039136 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 21:10:35.039411 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.039384 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" containerID="cri-o://975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520" gracePeriod=30 Nov 02 21:10:35.205789 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.204870 5136 patch_prober.go:29] interesting pod/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:35.205789 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.204955 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:35.348392 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.348309 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" event={"ID":"d072c3af-98c3-4a14-9810-c1385806e003","Type":"ContainerDied","Data":"1daadee9006b014e7eced0efe84fde18f63b7882d3783590b8e4f11848ce114e"} Nov 02 21:10:35.348392 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.348332 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll" Nov 02 21:10:35.348392 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.348370 5136 scope.go:122] "RemoveContainer" containerID="c1b81c27b1e9c64355b38fbe3682c7c573c3cb0d3a2aec59dd49fe44ee10576e" Nov 02 21:10:35.504521 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.504482 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 21:10:35.509827 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:35.509796 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-426941-76fddccf59-nn6ll"] Nov 02 21:10:36.546542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:36.546509 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d072c3af-98c3-4a14-9810-c1385806e003" path="/var/lib/kubelet/pods/d072c3af-98c3-4a14-9810-c1385806e003/volumes" Nov 02 21:10:37.457453 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:37.457422 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:37.457636 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:37.457472 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:40.307656 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.307622 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:10:40.358442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.358419 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 21:10:40.385198 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.385170 5136 generic.go:356] "Generic (PLEG): container finished" podID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerID="975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520" exitCode=0 Nov 02 21:10:40.385323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.385215 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" Nov 02 21:10:40.385323 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.385251 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" event={"ID":"02248d36-3a0a-40d1-8d1f-95c6198e4247","Type":"ContainerDied","Data":"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520"} Nov 02 21:10:40.385398 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.385333 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz" event={"ID":"02248d36-3a0a-40d1-8d1f-95c6198e4247","Type":"ContainerDied","Data":"8541208bcd6bb4e977bbffc8276ee801fa29fb8ef69e2f800e6ecea26656ecc0"} Nov 02 21:10:40.385398 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.385361 5136 scope.go:122] "RemoveContainer" containerID="975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520" Nov 02 21:10:40.387074 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387055 5136 generic.go:356] "Generic (PLEG): container finished" podID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerID="459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce" exitCode=0 Nov 02 21:10:40.387211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387084 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerDied","Data":"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce"} Nov 02 21:10:40.387211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387102 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" Nov 02 21:10:40.387211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387123 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg" event={"ID":"a73d6c30-028d-47cc-b37f-40227bda53f0","Type":"ContainerDied","Data":"c42436c480d91d9499ca34860c79d516ddffd367347a39c48087dd27d66ca872"} Nov 02 21:10:40.387584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387566 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387656 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387750 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387676 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387750 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387720 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387817 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387860 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387975 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387880 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387975 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387947 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.387975 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387969 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\") pod \"a73d6c30-028d-47cc-b37f-40227bda53f0\" (UID: \"a73d6c30-028d-47cc-b37f-40227bda53f0\") " Nov 02 21:10:40.388424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387964 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.388424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.387979 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.388644 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.388179 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.388644 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.388330 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.388644 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.388606 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.390254 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.390229 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.390459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.390437 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.390551 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.390529 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.390661 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.390645 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm" pod "a73d6c30-028d-47cc-b37f-40227bda53f0" (UID: "a73d6c30-028d-47cc-b37f-40227bda53f0"). InnerVolumeSpecName "kube-api-access-sw7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.398390 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.398369 5136 scope.go:122] "RemoveContainer" containerID="975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520" Nov 02 21:10:40.398698 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:40.398675 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520\": container with ID starting with 975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520 not found: ID does not exist" containerID="975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520" Nov 02 21:10:40.398782 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.398706 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520"} err="failed to get container status \"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520\": rpc error: code = NotFound desc = could not find container \"975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520\": container with ID starting with 975b6178eb9c43745bc892dddf59567d315054bb53c2b6ef5658679fda1e6520 not found: ID does not exist" Nov 02 21:10:40.398782 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.398724 5136 scope.go:122] "RemoveContainer" containerID="459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce" Nov 02 21:10:40.413244 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.413228 5136 scope.go:122] "RemoveContainer" containerID="d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a" Nov 02 21:10:40.423067 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.423048 5136 scope.go:122] "RemoveContainer" containerID="459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce" Nov 02 21:10:40.423362 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:40.423336 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce\": container with ID starting with 459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce not found: ID does not exist" containerID="459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce" Nov 02 21:10:40.423442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.423374 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce"} err="failed to get container status \"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce\": rpc error: code = NotFound desc = could not find container \"459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce\": container with ID starting with 459223f70b9e9be10cc49b8a7228d14be2ab1d0f6491abaf2a60eba201f0c8ce not found: ID does not exist" Nov 02 21:10:40.423442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.423397 5136 scope.go:122] "RemoveContainer" containerID="d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a" Nov 02 21:10:40.423704 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:40.423679 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a\": container with ID starting with d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a not found: ID does not exist" containerID="d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a" Nov 02 21:10:40.423769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.423714 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a"} err="failed to get container status \"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a\": rpc error: code = NotFound desc = could not find container \"d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a\": container with ID starting with d5581fc8cc7dcba7e335c1217a96b09f2fb09fd3aa65f7a134b5bf432940002a not found: ID does not exist" Nov 02 21:10:40.492174 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492139 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492351 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492215 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492456 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492349 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492456 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492400 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492456 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492451 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492509 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492493 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.492632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492542 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492598 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.492942 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.492698 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\") pod \"02248d36-3a0a-40d1-8d1f-95c6198e4247\" (UID: \"02248d36-3a0a-40d1-8d1f-95c6198e4247\") " Nov 02 21:10:40.493219 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493200 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73d6c30-028d-47cc-b37f-40227bda53f0-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493224 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw7rm\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-kube-api-access-sw7rm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493240 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493236 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493253 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493266 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493280 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493294 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73d6c30-028d-47cc-b37f-40227bda53f0-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493308 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493322 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73d6c30-028d-47cc-b37f-40227bda53f0-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493336 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.493844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493527 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.493844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.493791 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.495685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.495412 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.496101 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.496063 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.496600 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.496579 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "kube-api-access-l68rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.496999 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.496970 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.497858 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.497833 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token" pod "02248d36-3a0a-40d1-8d1f-95c6198e4247" (UID: "02248d36-3a0a-40d1-8d1f-95c6198e4247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.595938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.595533 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.595949 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.595965 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02248d36-3a0a-40d1-8d1f-95c6198e4247-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.595979 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.595991 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.596005 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.596019 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02248d36-3a0a-40d1-8d1f-95c6198e4247-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.596127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.596032 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68rk\" (UniqueName: \"kubernetes.io/projected/02248d36-3a0a-40d1-8d1f-95c6198e4247-kube-api-access-l68rk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.801359 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.799477 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 21:10:40.803342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.803314 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mtzhz"] Nov 02 21:10:40.906643 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.906556 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:10:40.911512 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:40.911474 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-68a6c923-openshift-default-77c75dd477-ltjfg"] Nov 02 21:10:41.814615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:41.814579 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:10:42.546945 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:42.546912 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" path="/var/lib/kubelet/pods/02248d36-3a0a-40d1-8d1f-95c6198e4247/volumes" Nov 02 21:10:42.547419 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:42.547405 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" path="/var/lib/kubelet/pods/a73d6c30-028d-47cc-b37f-40227bda53f0/volumes" Nov 02 21:10:44.687759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.687675 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 21:10:44.848285 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.848255 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\" (UniqueName: \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\") pod \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " Nov 02 21:10:44.848474 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.848462 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\") pod \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\" (UID: \"2b019b9f-ba31-4ce5-8b5f-e04e35af4512\") " Nov 02 21:10:44.848843 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.848816 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume" pod "2b019b9f-ba31-4ce5-8b5f-e04e35af4512" (UID: "2b019b9f-ba31-4ce5-8b5f-e04e35af4512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:44.850341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.850317 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8" pod "2b019b9f-ba31-4ce5-8b5f-e04e35af4512" (UID: "2b019b9f-ba31-4ce5-8b5f-e04e35af4512"). InnerVolumeSpecName "kube-api-access-ksgl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:44.950631 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.950602 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:44.950631 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:44.950633 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksgl8\" (UniqueName: \"kubernetes.io/projected/2b019b9f-ba31-4ce5-8b5f-e04e35af4512-kube-api-access-ksgl8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:45.426008 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.425928 5136 generic.go:356] "Generic (PLEG): container finished" podID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerID="23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572" exitCode=0 Nov 02 21:10:45.426008 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.425974 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" Nov 02 21:10:45.426233 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.426015 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" event={"ID":"2b019b9f-ba31-4ce5-8b5f-e04e35af4512","Type":"ContainerDied","Data":"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572"} Nov 02 21:10:45.426233 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.426071 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h" event={"ID":"2b019b9f-ba31-4ce5-8b5f-e04e35af4512","Type":"ContainerDied","Data":"37873c8cbc3961d33d910fdee0971d763164f3452c27e2dc5c792506d46db6d4"} Nov 02 21:10:45.426233 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.426098 5136 scope.go:122] "RemoveContainer" containerID="23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572" Nov 02 21:10:45.436520 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.436503 5136 scope.go:122] "RemoveContainer" containerID="23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572" Nov 02 21:10:45.436797 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:45.436778 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572\": container with ID starting with 23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572 not found: ID does not exist" containerID="23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572" Nov 02 21:10:45.436883 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.436806 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572"} err="failed to get container status \"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572\": rpc error: code = NotFound desc = could not find container \"23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572\": container with ID starting with 23534eecc4d9ffbcdc29001ea50148a14fe5bf0662eb7b2df44552f8ddbe1572 not found: ID does not exist" Nov 02 21:10:45.581300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.581271 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 21:10:45.628902 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:45.628873 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-426941-5b9978f65f-9qn7h"] Nov 02 21:10:46.549080 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:46.549048 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" path="/var/lib/kubelet/pods/2b019b9f-ba31-4ce5-8b5f-e04e35af4512/volumes" Nov 02 21:10:49.190291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:49.190253 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:10:51.013167 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:10:51.013129 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:10:58.875903 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:10:58.875873 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:11:04.211312 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.211272 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" podUID="22fdb42e-0ce5-4f59-9921-09a021a30af5" containerName="main" containerID="cri-o://f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b" gracePeriod=45 Nov 02 21:11:04.482392 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.482366 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:11:04.518621 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518595 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.518834 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518639 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.518834 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518694 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.518834 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518805 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.519044 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518946 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\" (UniqueName: \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.519044 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518967 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:04.519044 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518983 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:04.519044 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.518989 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\" (UniqueName: \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\") pod \"22fdb42e-0ce5-4f59-9921-09a021a30af5\" (UID: \"22fdb42e-0ce5-4f59-9921-09a021a30af5\") " Nov 02 21:11:04.519204 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.519035 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:04.519504 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.519483 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.519592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.519509 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.519592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.519524 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.521159 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.521135 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:04.521260 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.521173 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:04.521260 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.521195 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh" pod "22fdb42e-0ce5-4f59-9921-09a021a30af5" (UID: "22fdb42e-0ce5-4f59-9921-09a021a30af5"). InnerVolumeSpecName "kube-api-access-5q8gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:04.551664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.551635 5136 generic.go:356] "Generic (PLEG): container finished" podID="22fdb42e-0ce5-4f59-9921-09a021a30af5" containerID="f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b" exitCode=0 Nov 02 21:11:04.551844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.551674 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" Nov 02 21:11:04.551844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.551716 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" event={"ID":"22fdb42e-0ce5-4f59-9921-09a021a30af5","Type":"ContainerDied","Data":"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b"} Nov 02 21:11:04.551844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.551776 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j" event={"ID":"22fdb42e-0ce5-4f59-9921-09a021a30af5","Type":"ContainerDied","Data":"21c5881cd77c3e077e206c2f911a8174514bb69b44d7f1835c9647f787a18b61"} Nov 02 21:11:04.551844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.551798 5136 scope.go:122] "RemoveContainer" containerID="f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b" Nov 02 21:11:04.562397 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.562372 5136 scope.go:122] "RemoveContainer" containerID="f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b" Nov 02 21:11:04.562691 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:04.562670 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b\": container with ID starting with f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b not found: ID does not exist" containerID="f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b" Nov 02 21:11:04.562795 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.562700 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b"} err="failed to get container status \"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b\": rpc error: code = NotFound desc = could not find container \"f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b\": container with ID starting with f0dba8d47dc3cf34860c978ce44423abce49534fa1e54d71b2bddc97db9f451b not found: ID does not exist" Nov 02 21:11:04.621330 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.621298 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22fdb42e-0ce5-4f59-9921-09a021a30af5-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.621582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.621560 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8gh\" (UniqueName: \"kubernetes.io/projected/22fdb42e-0ce5-4f59-9921-09a021a30af5-kube-api-access-5q8gh\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.621582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.621582 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fdb42e-0ce5-4f59-9921-09a021a30af5-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:04.698684 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.698650 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:11:04.701175 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:04.701144 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/rate-test-model-76c687-kserve-7c95cfd957-bqd9j"] Nov 02 21:11:06.547104 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:06.547072 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fdb42e-0ce5-4f59-9921-09a021a30af5" path="/var/lib/kubelet/pods/22fdb42e-0ce5-4f59-9921-09a021a30af5/volumes" Nov 02 21:11:10.593873 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.593842 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 21:11:10.594330 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.594044 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" podUID="54fd3124-1e2d-443d-82c1-70084f1e299d" containerName="maas-api" containerID="cri-o://86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc" gracePeriod=30 Nov 02 21:11:10.886854 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.886828 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 21:11:10.999752 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.996065 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\") pod \"54fd3124-1e2d-443d-82c1-70084f1e299d\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " Nov 02 21:11:10.999752 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.996204 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\" (UniqueName: \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\") pod \"54fd3124-1e2d-443d-82c1-70084f1e299d\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " Nov 02 21:11:10.999752 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.996279 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") pod \"54fd3124-1e2d-443d-82c1-70084f1e299d\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " Nov 02 21:11:10.999752 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:10.996318 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\") pod \"54fd3124-1e2d-443d-82c1-70084f1e299d\" (UID: \"54fd3124-1e2d-443d-82c1-70084f1e299d\") " Nov 02 21:11:11.005759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.004072 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca" pod "54fd3124-1e2d-443d-82c1-70084f1e299d" (UID: "54fd3124-1e2d-443d-82c1-70084f1e299d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:11.008290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.008208 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls" pod "54fd3124-1e2d-443d-82c1-70084f1e299d" (UID: "54fd3124-1e2d-443d-82c1-70084f1e299d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:11.010120 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.010094 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n" pod "54fd3124-1e2d-443d-82c1-70084f1e299d" (UID: "54fd3124-1e2d-443d-82c1-70084f1e299d"). InnerVolumeSpecName "kube-api-access-vwt7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:11.018177 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.018145 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs" pod "54fd3124-1e2d-443d-82c1-70084f1e299d" (UID: "54fd3124-1e2d-443d-82c1-70084f1e299d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:11.098366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.098336 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwt7n\" (UniqueName: \"kubernetes.io/projected/54fd3124-1e2d-443d-82c1-70084f1e299d-kube-api-access-vwt7n\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:11.098366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.098367 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:11.098523 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.098382 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54fd3124-1e2d-443d-82c1-70084f1e299d-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:11.098523 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.098394 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54fd3124-1e2d-443d-82c1-70084f1e299d-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:11.611036 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.610996 5136 generic.go:356] "Generic (PLEG): container finished" podID="54fd3124-1e2d-443d-82c1-70084f1e299d" containerID="86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc" exitCode=0 Nov 02 21:11:11.611550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.611067 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" Nov 02 21:11:11.611550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.611086 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" event={"ID":"54fd3124-1e2d-443d-82c1-70084f1e299d","Type":"ContainerDied","Data":"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc"} Nov 02 21:11:11.611550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.611166 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc" event={"ID":"54fd3124-1e2d-443d-82c1-70084f1e299d","Type":"ContainerDied","Data":"c87f5beed816ce561d5deed7b26b337ccdb17981fec24b5079144aad1973d99e"} Nov 02 21:11:11.611550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.611200 5136 scope.go:122] "RemoveContainer" containerID="86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc" Nov 02 21:11:11.622797 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.622776 5136 scope.go:122] "RemoveContainer" containerID="86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc" Nov 02 21:11:11.623079 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:11.623056 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc\": container with ID starting with 86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc not found: ID does not exist" containerID="86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc" Nov 02 21:11:11.623152 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.623093 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc"} err="failed to get container status \"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc\": rpc error: code = NotFound desc = could not find container \"86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc\": container with ID starting with 86793c3a573787b50ad0ab11d136890c7a8d7222f52f8a09f7a197668102e5cc not found: ID does not exist" Nov 02 21:11:11.766739 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.766692 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 21:11:11.814137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:11.814099 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a4ab0-68fdf96674-t7prc"] Nov 02 21:11:12.546849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:12.546820 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fd3124-1e2d-443d-82c1-70084f1e299d" path="/var/lib/kubelet/pods/54fd3124-1e2d-443d-82c1-70084f1e299d/volumes" Nov 02 21:11:13.897419 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:13.897379 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" podUID="ddb73808-afeb-4621-860c-8b7472ee389b" containerName="main" containerID="cri-o://8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251" gracePeriod=45 Nov 02 21:11:14.174416 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.174387 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:11:14.233061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233027 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233243 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233160 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233243 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233197 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233378 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233277 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233378 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233291 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:14.233378 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233360 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\" (UniqueName: \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233563 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233405 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\") pod \"ddb73808-afeb-4621-860c-8b7472ee389b\" (UID: \"ddb73808-afeb-4621-860c-8b7472ee389b\") " Nov 02 21:11:14.233563 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233406 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:14.233695 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233652 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:14.233951 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233924 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.233951 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233949 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.234150 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.233960 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.235573 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.235547 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "kube-api-access-gzqfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:14.235669 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.235577 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:14.235669 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.235592 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm" pod "ddb73808-afeb-4621-860c-8b7472ee389b" (UID: "ddb73808-afeb-4621-860c-8b7472ee389b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:14.334944 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.334900 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb73808-afeb-4621-860c-8b7472ee389b-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.334944 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.334942 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb73808-afeb-4621-860c-8b7472ee389b-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.334944 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.334955 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqfm\" (UniqueName: \"kubernetes.io/projected/ddb73808-afeb-4621-860c-8b7472ee389b-kube-api-access-gzqfm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:14.633900 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.633869 5136 generic.go:356] "Generic (PLEG): container finished" podID="ddb73808-afeb-4621-860c-8b7472ee389b" containerID="8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251" exitCode=0 Nov 02 21:11:14.634093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.633912 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" Nov 02 21:11:14.634093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.633958 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" event={"ID":"ddb73808-afeb-4621-860c-8b7472ee389b","Type":"ContainerDied","Data":"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251"} Nov 02 21:11:14.634093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.634003 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn" event={"ID":"ddb73808-afeb-4621-860c-8b7472ee389b","Type":"ContainerDied","Data":"8009636646a7ceaf3dcd828e5d907083a6fe6617e1751489780669eb8c4a4d01"} Nov 02 21:11:14.634093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.634024 5136 scope.go:122] "RemoveContainer" containerID="8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251" Nov 02 21:11:14.643809 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.643787 5136 scope.go:122] "RemoveContainer" containerID="8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251" Nov 02 21:11:14.644079 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:14.644057 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251\": container with ID starting with 8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251 not found: ID does not exist" containerID="8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251" Nov 02 21:11:14.644137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.644087 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251"} err="failed to get container status \"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251\": rpc error: code = NotFound desc = could not find container \"8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251\": container with ID starting with 8fd18f13faca2ca97fbd81e325e8e424a763e8c05f732996250541c94f3fe251 not found: ID does not exist" Nov 02 21:11:14.789535 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.789494 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:11:14.837190 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:14.837105 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/rate-test-model-6b12ae-kserve-75c79657d5-5rvfn"] Nov 02 21:11:16.547554 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:16.547475 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb73808-afeb-4621-860c-8b7472ee389b" path="/var/lib/kubelet/pods/ddb73808-afeb-4621-860c-8b7472ee389b/volumes" Nov 02 21:11:16.882366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:16.882285 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:11:21.012665 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:21.012632 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:11:21.403638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:21.403545 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 21:11:21.403849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:21.403827 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" containerID="cri-o://d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f" gracePeriod=30 Nov 02 21:11:26.675264 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.675197 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 21:11:26.713299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.713266 5136 generic.go:356] "Generic (PLEG): container finished" podID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerID="d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f" exitCode=0 Nov 02 21:11:26.713444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.713324 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerDied","Data":"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f"} Nov 02 21:11:26.713444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.713330 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" Nov 02 21:11:26.713444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.713373 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp" event={"ID":"ddd213e5-e8ba-4aac-b235-5dc90614d08d","Type":"ContainerDied","Data":"7fed349ea7a620f0fb843320aff04a6bee91dba4306ac0c3557cf89cef480c31"} Nov 02 21:11:26.713444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.713396 5136 scope.go:122] "RemoveContainer" containerID="d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f" Nov 02 21:11:26.723414 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.723385 5136 scope.go:122] "RemoveContainer" containerID="5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997" Nov 02 21:11:26.732603 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.732582 5136 scope.go:122] "RemoveContainer" containerID="d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f" Nov 02 21:11:26.732994 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:26.732976 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f\": container with ID starting with d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f not found: ID does not exist" containerID="d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f" Nov 02 21:11:26.733053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.733004 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f"} err="failed to get container status \"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f\": rpc error: code = NotFound desc = could not find container \"d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f\": container with ID starting with d420bdf5b6771e92f8bf3b9ceb0185d3a8ecf84d988733f30ed5e3d964f0aa5f not found: ID does not exist" Nov 02 21:11:26.733053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.733022 5136 scope.go:122] "RemoveContainer" containerID="5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997" Nov 02 21:11:26.733289 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:26.733267 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997\": container with ID starting with 5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997 not found: ID does not exist" containerID="5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997" Nov 02 21:11:26.733342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.733294 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997"} err="failed to get container status \"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997\": rpc error: code = NotFound desc = could not find container \"5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997\": container with ID starting with 5d890bb899a90769dff566c42d7ad08d450e51fc59dbd3d86cc455edb08c7997 not found: ID does not exist" Nov 02 21:11:26.790397 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790358 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790410 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790484 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790535 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790562 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790581 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790642 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790680 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790702 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:26.790965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790721 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\") pod \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\" (UID: \"ddd213e5-e8ba-4aac-b235-5dc90614d08d\") " Nov 02 21:11:26.790965 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790867 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:26.791229 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.790984 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:26.791229 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791186 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:26.791339 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791236 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:26.791339 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791332 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.791457 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791352 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.791457 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791366 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.791457 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791375 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.791457 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.791385 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.792931 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.792906 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:26.792931 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.792909 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "kube-api-access-pxk62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:26.793200 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.793177 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:26.793270 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.793184 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo" pod "ddd213e5-e8ba-4aac-b235-5dc90614d08d" (UID: "ddd213e5-e8ba-4aac-b235-5dc90614d08d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:11:26.892374 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.892339 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-kube-api-access-pxk62\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.892374 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.892369 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.892374 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.892379 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:26.892628 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:26.892388 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd213e5-e8ba-4aac-b235-5dc90614d08d-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:27.129501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:27.129449 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 21:11:27.133265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:27.133237 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-f2krp"] Nov 02 21:11:28.545966 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:28.545932 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" path="/var/lib/kubelet/pods/ddd213e5-e8ba-4aac-b235-5dc90614d08d/volumes" Nov 02 21:11:31.903598 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:31.903530 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerName="main" containerID="cri-o://02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc" gracePeriod=45 Nov 02 21:11:32.060030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.059981 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": dial tcp 10.133.0.193:8000: connect: connection refused" Nov 02 21:11:32.174877 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.174847 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:11:32.254246 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254212 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254261 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\" (UniqueName: \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254320 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254388 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254433 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254484 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:32.254665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254629 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\") pod \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\" (UID: \"baa2139e-b3c4-4f3d-8cb6-62437d5203fb\") " Nov 02 21:11:32.254665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254659 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:32.254845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.254767 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:32.255075 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.255058 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.255158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.255077 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.255158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.255111 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.256709 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.256680 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:32.256709 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.256682 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "kube-api-access-zx84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:32.256880 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.256708 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs" pod "baa2139e-b3c4-4f3d-8cb6-62437d5203fb" (UID: "baa2139e-b3c4-4f3d-8cb6-62437d5203fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:32.357670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.357627 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.357670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.357672 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx84n\" (UniqueName: \"kubernetes.io/projected/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-kube-api-access-zx84n\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.357992 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.357691 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baa2139e-b3c4-4f3d-8cb6-62437d5203fb-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:32.756749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.752042 5136 generic.go:356] "Generic (PLEG): container finished" podID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerID="02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc" exitCode=0 Nov 02 21:11:32.756749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.752111 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" Nov 02 21:11:32.756749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.752125 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" event={"ID":"baa2139e-b3c4-4f3d-8cb6-62437d5203fb","Type":"ContainerDied","Data":"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc"} Nov 02 21:11:32.756749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.752184 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj" event={"ID":"baa2139e-b3c4-4f3d-8cb6-62437d5203fb","Type":"ContainerDied","Data":"0329769e8a216c6136b3c1abfa671a91c1a4142061f0980a71fbf554aea72ec1"} Nov 02 21:11:32.756749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.752203 5136 scope.go:122] "RemoveContainer" containerID="02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc" Nov 02 21:11:32.764651 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.764632 5136 scope.go:122] "RemoveContainer" containerID="02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc" Nov 02 21:11:32.764994 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:32.764975 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc\": container with ID starting with 02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc not found: ID does not exist" containerID="02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc" Nov 02 21:11:32.765077 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.765007 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc"} err="failed to get container status \"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc\": rpc error: code = NotFound desc = could not find container \"02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc\": container with ID starting with 02fb946455fbe0264e36574de40104dd18023d25ad95450f94182f7024c714bc not found: ID does not exist" Nov 02 21:11:32.918639 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.918538 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:11:32.985460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:32.985419 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/sub-test-model-a9ed60-kserve-c8cbf94ff-z6kkj"] Nov 02 21:11:34.548012 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:34.547976 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" path="/var/lib/kubelet/pods/baa2139e-b3c4-4f3d-8cb6-62437d5203fb/volumes" Nov 02 21:11:50.185700 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.185665 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:11:50.186206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186190 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" Nov 02 21:11:50.186257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186207 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" Nov 02 21:11:50.186257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186225 5136 cpu_manager.go:414] "containerMap: removing container" podUID="d072c3af-98c3-4a14-9810-c1385806e003" containerName="maas-api" Nov 02 21:11:50.186257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186233 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d072c3af-98c3-4a14-9810-c1385806e003" containerName="maas-api" Nov 02 21:11:50.186257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186244 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:11:50.186257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186251 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186282 5136 cpu_manager.go:414] "containerMap: removing container" podUID="22fdb42e-0ce5-4f59-9921-09a021a30af5" containerName="main" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186288 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fdb42e-0ce5-4f59-9921-09a021a30af5" containerName="main" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186302 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186307 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186316 5136 cpu_manager.go:414] "containerMap: removing container" podUID="54fd3124-1e2d-443d-82c1-70084f1e299d" containerName="maas-api" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186321 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fd3124-1e2d-443d-82c1-70084f1e299d" containerName="maas-api" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186328 5136 cpu_manager.go:414] "containerMap: removing container" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerName="main" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186332 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerName="main" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186339 5136 cpu_manager.go:414] "containerMap: removing container" podUID="965a253c-ede8-4374-b612-491c18c0461c" containerName="maas-api" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186343 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965a253c-ede8-4374-b612-491c18c0461c" containerName="maas-api" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186354 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186359 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186367 5136 cpu_manager.go:414] "containerMap: removing container" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186371 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186381 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186385 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186402 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ddb73808-afeb-4621-860c-8b7472ee389b" containerName="main" Nov 02 21:11:50.186418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186407 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb73808-afeb-4621-860c-8b7472ee389b" containerName="main" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186520 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fdb42e-0ce5-4f59-9921-09a021a30af5" containerName="main" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186538 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b019b9f-ba31-4ce5-8b5f-e04e35af4512" containerName="payload-pre-processing" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186550 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fd3124-1e2d-443d-82c1-70084f1e299d" containerName="maas-api" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186559 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa2139e-b3c4-4f3d-8cb6-62437d5203fb" containerName="main" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186568 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="965a253c-ede8-4374-b612-491c18c0461c" containerName="maas-api" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186579 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186587 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="02248d36-3a0a-40d1-8d1f-95c6198e4247" containerName="istio-proxy" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186597 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73d6c30-028d-47cc-b37f-40227bda53f0" containerName="istio-proxy" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186608 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186617 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb73808-afeb-4621-860c-8b7472ee389b" containerName="main" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186625 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4053b5f-e54a-405a-bbf9-bd67a1650d4a" containerName="main" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186635 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="d072c3af-98c3-4a14-9810-c1385806e003" containerName="maas-api" Nov 02 21:11:50.187013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.186641 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:11:50.190136 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.190117 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.243849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.243812 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:11:50.283109 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283078 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.283283 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283189 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.283283 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283238 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.283283 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283261 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.283478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283375 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.283478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.283433 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxb5b\" (UniqueName: \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.288007 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.287984 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"openshift-service-ca.crt\"" Nov 02 21:11:50.288007 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.287989 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"kube-root-ca.crt\"" Nov 02 21:11:50.288173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.288034 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"sub-test-model-76c687-kserve-self-signed-certs\"" Nov 02 21:11:50.288173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.288075 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-76c687\"/\"default-dockercfg-c8gsb\"" Nov 02 21:11:50.385039 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.384999 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385078 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385095 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385116 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385159 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385492 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385275 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxb5b\" (UniqueName: \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385622 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385598 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385699 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385627 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.385699 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.385638 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.387744 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.387699 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.387913 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.387895 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.439860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.439807 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxb5b\" (UniqueName: \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\") pod \"sub-test-model-76c687-kserve-7b5bdf459-xjc6n\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.503593 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.503567 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:50.874538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:50.874511 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:11:50.876010 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:11:50.875979 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be482bd_db18_4449_b943_8c2d93236048.slice/crio-045d0e119b306f285008516c25c51ec3878c4661d0c3acce420e68053e3c460d WatchSource:0}: Error finding container 045d0e119b306f285008516c25c51ec3878c4661d0c3acce420e68053e3c460d: Status 404 returned error can't find the container with id 045d0e119b306f285008516c25c51ec3878c4661d0c3acce420e68053e3c460d Nov 02 21:11:51.003786 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:11:51.003724 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:11:51.870143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:51.870107 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" event={"ID":"9be482bd-db18-4449-b943-8c2d93236048","Type":"ContainerStarted","Data":"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e"} Nov 02 21:11:51.870570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:51.870154 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" event={"ID":"9be482bd-db18-4449-b943-8c2d93236048","Type":"ContainerStarted","Data":"045d0e119b306f285008516c25c51ec3878c4661d0c3acce420e68053e3c460d"} Nov 02 21:11:51.980584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:51.980529 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" podStartSLOduration=1.980515486 podStartE2EDuration="1.980515486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:11:51.978047353 +0000 UTC m=+4995.584591851" watchObservedRunningTime="2026-11-02 21:11:51.980515486 +0000 UTC m=+4995.587059984" Nov 02 21:11:52.504548 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:52.504517 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:11:52.700682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:11:52.700650 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:12:00.504173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:00.504137 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:12:00.516325 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:00.516301 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:12:00.934698 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:00.934616 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:12:07.722478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.722431 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" podUID="2796ae70-7b8c-43d1-8122-e06c738fd254" containerName="main" containerID="cri-o://d999752d889c36295a46044db8f810aea01f2027f178bd071e86d1c68557db71" gracePeriod=45 Nov 02 21:12:07.965271 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.965240 5136 generic.go:356] "Generic (PLEG): container finished" podID="2796ae70-7b8c-43d1-8122-e06c738fd254" containerID="d999752d889c36295a46044db8f810aea01f2027f178bd071e86d1c68557db71" exitCode=0 Nov 02 21:12:07.965403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.965308 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" event={"ID":"2796ae70-7b8c-43d1-8122-e06c738fd254","Type":"ContainerDied","Data":"d999752d889c36295a46044db8f810aea01f2027f178bd071e86d1c68557db71"} Nov 02 21:12:07.965403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.965353 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" event={"ID":"2796ae70-7b8c-43d1-8122-e06c738fd254","Type":"ContainerDied","Data":"a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419"} Nov 02 21:12:07.965403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.965366 5136 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a672fee462f1bde810319f85bdf242751db0020e8b7086479d56bf6de4507419" Nov 02 21:12:07.980188 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:07.980138 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:12:08.096983 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.096950 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\" (UniqueName: \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097164 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097003 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097164 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097111 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\" (UniqueName: \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097199 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097231 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097259 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\") pod \"2796ae70-7b8c-43d1-8122-e06c738fd254\" (UID: \"2796ae70-7b8c-43d1-8122-e06c738fd254\") " Nov 02 21:12:08.097475 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097330 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:08.097570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097551 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:08.097778 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097754 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:08.097896 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097879 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.097961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097903 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.097961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.097917 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.101495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.101474 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:12:08.101625 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.101491 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:08.101625 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.101501 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x" pod "2796ae70-7b8c-43d1-8122-e06c738fd254" (UID: "2796ae70-7b8c-43d1-8122-e06c738fd254"). InnerVolumeSpecName "kube-api-access-g9t6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:08.199466 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.199430 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796ae70-7b8c-43d1-8122-e06c738fd254-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.199466 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.199457 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9t6x\" (UniqueName: \"kubernetes.io/projected/2796ae70-7b8c-43d1-8122-e06c738fd254-kube-api-access-g9t6x\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.199466 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.199468 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796ae70-7b8c-43d1-8122-e06c738fd254-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:08.970827 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:08.970794 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2" Nov 02 21:12:09.128328 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:09.128276 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:12:09.130469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:09.130445 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-578636/sub-test-model-578636-kserve-67895489f9-jp5b2"] Nov 02 21:12:10.546696 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:10.546652 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2796ae70-7b8c-43d1-8122-e06c738fd254" path="/var/lib/kubelet/pods/2796ae70-7b8c-43d1-8122-e06c738fd254/volumes" Nov 02 21:12:21.004339 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:12:21.004305 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:12:25.278411 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:25.278378 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 21:12:25.278976 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:25.278622 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" containerID="cri-o://f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6" gracePeriod=30 Nov 02 21:12:30.538773 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.538747 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 21:12:30.571377 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571317 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571377 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571350 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571391 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571409 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571456 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571493 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571556 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571596 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571597 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:30.571866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571744 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\") pod \"0aae6438-56d6-4650-ab1b-97e08f1cded8\" (UID: \"0aae6438-56d6-4650-ab1b-97e08f1cded8\") " Nov 02 21:12:30.571866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571841 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:30.572050 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.571859 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:12:30.572127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572077 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:30.572127 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572099 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:30.572274 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572256 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.572342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572275 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae6438-56d6-4650-ab1b-97e08f1cded8-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.572342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572293 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.572342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572307 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.572342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.572321 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.573796 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.573774 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:12:30.573899 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.573874 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "kube-api-access-j4lbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:30.574173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.574152 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:12:30.574235 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.574166 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token" pod "0aae6438-56d6-4650-ab1b-97e08f1cded8" (UID: "0aae6438-56d6-4650-ab1b-97e08f1cded8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:30.673495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.673467 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.673495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.673497 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.673677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.673511 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae6438-56d6-4650-ab1b-97e08f1cded8-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:30.673677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:30.673521 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lbk\" (UniqueName: \"kubernetes.io/projected/0aae6438-56d6-4650-ab1b-97e08f1cded8-kube-api-access-j4lbk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:31.098307 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.098270 5136 generic.go:356] "Generic (PLEG): container finished" podID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerID="f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6" exitCode=0 Nov 02 21:12:31.098487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.098346 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" Nov 02 21:12:31.098487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.098358 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerDied","Data":"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6"} Nov 02 21:12:31.098487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.098418 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26" event={"ID":"0aae6438-56d6-4650-ab1b-97e08f1cded8","Type":"ContainerDied","Data":"fef50137a41abd08033b4c495ca0065df73106aa3b817283b408468680756d78"} Nov 02 21:12:31.098487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.098441 5136 scope.go:122] "RemoveContainer" containerID="f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6" Nov 02 21:12:31.108547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.108529 5136 scope.go:122] "RemoveContainer" containerID="42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749" Nov 02 21:12:31.117999 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.117972 5136 scope.go:122] "RemoveContainer" containerID="f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6" Nov 02 21:12:31.118268 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:12:31.118247 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6\": container with ID starting with f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6 not found: ID does not exist" containerID="f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6" Nov 02 21:12:31.118315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.118279 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6"} err="failed to get container status \"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6\": rpc error: code = NotFound desc = could not find container \"f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6\": container with ID starting with f38d3658d991c17603006bc6a8e2f4926e651864b431df448e2e54563568d0e6 not found: ID does not exist" Nov 02 21:12:31.118315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.118296 5136 scope.go:122] "RemoveContainer" containerID="42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749" Nov 02 21:12:31.118530 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:12:31.118508 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749\": container with ID starting with 42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749 not found: ID does not exist" containerID="42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749" Nov 02 21:12:31.118624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.118534 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749"} err="failed to get container status \"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749\": rpc error: code = NotFound desc = could not find container \"42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749\": container with ID starting with 42b1aa52dcc2b05f16d76f32c684081467c59081927c854c22fb7c9680389749 not found: ID does not exist" Nov 02 21:12:31.256441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.256407 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 21:12:31.261710 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.261685 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9ed60-openshift-default-b6546977-w6j26"] Nov 02 21:12:31.378961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:31.378889 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 21:12:32.546502 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:32.546466 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" path="/var/lib/kubelet/pods/0aae6438-56d6-4650-ab1b-97e08f1cded8/volumes" Nov 02 21:12:36.379616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:36.379576 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" containerID="cri-o://73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5" gracePeriod=25 Nov 02 21:12:38.277032 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:38.277001 5136 scope.go:122] "RemoveContainer" containerID="d999752d889c36295a46044db8f810aea01f2027f178bd071e86d1c68557db71" Nov 02 21:12:43.877656 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:43.877617 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:51.003943 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:12:51.003859 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:12:53.878112 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:53.878072 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:56.640314 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.640260 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 21:12:56.684018 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.683987 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\") pod \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " Nov 02 21:12:56.684192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.684151 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\" (UniqueName: \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\") pod \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\" (UID: \"88aaf7ec-3461-4e97-9a5a-dec550bc45ec\") " Nov 02 21:12:56.684434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.684405 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume" pod "88aaf7ec-3461-4e97-9a5a-dec550bc45ec" (UID: "88aaf7ec-3461-4e97-9a5a-dec550bc45ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:12:56.684634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.684615 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:56.686241 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.686212 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc" pod "88aaf7ec-3461-4e97-9a5a-dec550bc45ec" (UID: "88aaf7ec-3461-4e97-9a5a-dec550bc45ec"). InnerVolumeSpecName "kube-api-access-gw7qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:56.785595 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:56.785541 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7qc\" (UniqueName: \"kubernetes.io/projected/88aaf7ec-3461-4e97-9a5a-dec550bc45ec-kube-api-access-gw7qc\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:57.248061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.248023 5136 generic.go:356] "Generic (PLEG): container finished" podID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerID="73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5" exitCode=0 Nov 02 21:12:57.248061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.248063 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" Nov 02 21:12:57.248308 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.248070 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" event={"ID":"88aaf7ec-3461-4e97-9a5a-dec550bc45ec","Type":"ContainerDied","Data":"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5"} Nov 02 21:12:57.248308 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.248123 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m" event={"ID":"88aaf7ec-3461-4e97-9a5a-dec550bc45ec","Type":"ContainerDied","Data":"24634f51669f8a85d1816aba334a9a440c03e8f8a39440960aafcf3d9429a6b8"} Nov 02 21:12:57.248308 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.248146 5136 scope.go:122] "RemoveContainer" containerID="73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5" Nov 02 21:12:57.261070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.261031 5136 scope.go:122] "RemoveContainer" containerID="73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5" Nov 02 21:12:57.261309 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:12:57.261292 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5\": container with ID starting with 73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5 not found: ID does not exist" containerID="73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5" Nov 02 21:12:57.261362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.261317 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5"} err="failed to get container status \"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5\": rpc error: code = NotFound desc = could not find container \"73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5\": container with ID starting with 73d4869ba86543ff2f96be3effcc0bb3accc848a6def2c106ce1621ea156eec5 not found: ID does not exist" Nov 02 21:12:57.406234 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.406201 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 21:12:57.453540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:57.453507 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-578636-54885c69f5-9wq2m"] Nov 02 21:12:58.547142 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:12:58.547104 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" path="/var/lib/kubelet/pods/88aaf7ec-3461-4e97-9a5a-dec550bc45ec/volumes" Nov 02 21:13:12.249400 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:12.249366 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:13:12.249910 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:12.249658 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" containerID="cri-o://3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0" gracePeriod=30 Nov 02 21:13:12.431906 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:12.431864 5136 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:13:12.432092 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:12.431953 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:13:17.508095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.508069 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:13:17.632680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632608 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632721 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632762 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632784 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632807 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632826 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.632928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632858 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632930 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633053 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633159 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.632931 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633237 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.633286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633237 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:17.633619 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633296 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\") pod \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\" (UID: \"10baf3a1-6efd-4f99-8cd0-87a3a105666d\") " Nov 02 21:13:17.633762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633709 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.633762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633761 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.633979 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633771 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.633979 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633780 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.633979 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.633789 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.635155 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.635129 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:13:17.635367 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.635349 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:17.635524 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.635496 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.635524 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.635514 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76" pod "10baf3a1-6efd-4f99-8cd0-87a3a105666d" (UID: "10baf3a1-6efd-4f99-8cd0-87a3a105666d"). InnerVolumeSpecName "kube-api-access-rfn76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:17.722366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.722336 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 21:13:17.734997 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.734968 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.735132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.735001 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.735132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.735017 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfn76\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-kube-api-access-rfn76\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.735132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:17.735032 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10baf3a1-6efd-4f99-8cd0-87a3a105666d-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:18.365707 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.365657 5136 generic.go:356] "Generic (PLEG): container finished" podID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerID="3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0" exitCode=0 Nov 02 21:13:18.365707 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.365705 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" Nov 02 21:13:18.366025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.365768 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" event={"ID":"10baf3a1-6efd-4f99-8cd0-87a3a105666d","Type":"ContainerDied","Data":"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0"} Nov 02 21:13:18.366025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.365833 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c" event={"ID":"10baf3a1-6efd-4f99-8cd0-87a3a105666d","Type":"ContainerDied","Data":"db9a40dcf73a2f101dbf704208dc84cddd5e406e85a15c24412ff17f5bce3b03"} Nov 02 21:13:18.366025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.365865 5136 scope.go:122] "RemoveContainer" containerID="3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0" Nov 02 21:13:18.377055 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.377029 5136 scope.go:122] "RemoveContainer" containerID="3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0" Nov 02 21:13:18.377387 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:13:18.377364 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0\": container with ID starting with 3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0 not found: ID does not exist" containerID="3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0" Nov 02 21:13:18.377534 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.377393 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0"} err="failed to get container status \"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0\": rpc error: code = NotFound desc = could not find container \"3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0\": container with ID starting with 3ad9e9915e27dbea1a80968e958b41c5423697198d109994400e0bfb5e891bd0 not found: ID does not exist" Nov 02 21:13:18.521743 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.521705 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:13:18.528079 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.528050 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-f6v2c"] Nov 02 21:13:18.546680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:18.546640 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" path="/var/lib/kubelet/pods/10baf3a1-6efd-4f99-8cd0-87a3a105666d/volumes" Nov 02 21:13:21.004262 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:13:21.004230 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:13:26.220644 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.220602 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:13:26.221396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221374 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221397 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd213e5-e8ba-4aac-b235-5dc90614d08d" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221417 5136 cpu_manager.go:414] "containerMap: removing container" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221424 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221444 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221452 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221478 5136 cpu_manager.go:414] "containerMap: removing container" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221486 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221509 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2796ae70-7b8c-43d1-8122-e06c738fd254" containerName="main" Nov 02 21:13:26.221531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221517 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2796ae70-7b8c-43d1-8122-e06c738fd254" containerName="main" Nov 02 21:13:26.222029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221641 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="88aaf7ec-3461-4e97-9a5a-dec550bc45ec" containerName="payload-pre-processing" Nov 02 21:13:26.222029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221655 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:13:26.222029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221670 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2796ae70-7b8c-43d1-8122-e06c738fd254" containerName="main" Nov 02 21:13:26.222029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.221686 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="10baf3a1-6efd-4f99-8cd0-87a3a105666d" containerName="istio-proxy" Nov 02 21:13:26.225150 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.225129 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231134 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231103 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcwj\" (UniqueName: \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231241 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231148 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231287 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231260 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231475 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231454 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231482 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.231513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.231500 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.322583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.322544 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"openshift-service-ca.crt\"" Nov 02 21:13:26.322861 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.322780 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"kube-root-ca.crt\"" Nov 02 21:13:26.322861 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.322802 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"sub-test-model-6b12ae-kserve-self-signed-certs\"" Nov 02 21:13:26.322861 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.322821 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6b12ae\"/\"default-dockercfg-kxl45\"" Nov 02 21:13:26.324932 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.324912 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:13:26.332398 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332376 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332510 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332477 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332510 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332496 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332512 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332535 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcwj\" (UniqueName: \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332552 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332894 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332867 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.332981 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332943 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.333043 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.332996 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.334797 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.334779 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.335067 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.335052 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.386310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.386281 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcwj\" (UniqueName: \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\") pod \"sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.537458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.537363 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:26.910251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:26.910224 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:13:26.911849 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:13:26.911814 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac130cd9_f3ab_4b01_9e38_7d7647f62468.slice/crio-be92f0a4f303c8b5d566b306fa33ba9ad94f4500833735ec4d150df1ed096d0f WatchSource:0}: Error finding container be92f0a4f303c8b5d566b306fa33ba9ad94f4500833735ec4d150df1ed096d0f: Status 404 returned error can't find the container with id be92f0a4f303c8b5d566b306fa33ba9ad94f4500833735ec4d150df1ed096d0f Nov 02 21:13:27.422027 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:27.421984 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" event={"ID":"ac130cd9-f3ab-4b01-9e38-7d7647f62468","Type":"ContainerStarted","Data":"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04"} Nov 02 21:13:27.422027 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:27.422034 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" event={"ID":"ac130cd9-f3ab-4b01-9e38-7d7647f62468","Type":"ContainerStarted","Data":"be92f0a4f303c8b5d566b306fa33ba9ad94f4500833735ec4d150df1ed096d0f"} Nov 02 21:13:27.530832 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:27.530775 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" podStartSLOduration=1.530759213 podStartE2EDuration="1.530759213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:27.529678512 +0000 UTC m=+5091.136223011" watchObservedRunningTime="2026-11-02 21:13:27.530759213 +0000 UTC m=+5091.137303711" Nov 02 21:13:27.537857 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:27.537831 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:32.754491 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:32.750399 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" containerID="cri-o://860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387" gracePeriod=45 Nov 02 21:13:32.758791 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:32.758412 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": EOF" Nov 02 21:13:33.031065 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.030859 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 21:13:33.109254 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109227 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.109459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109276 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\" (UniqueName: \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.109459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109337 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\" (UniqueName: \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.109459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109386 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.109459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109457 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.109780 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.109534 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\") pod \"42f903d4-b68e-4291-869f-2092f360a459\" (UID: \"42f903d4-b68e-4291-869f-2092f360a459\") " Nov 02 21:13:33.110450 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.110407 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.110787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.110687 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.110787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.110762 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.111320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.111301 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.111449 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.111434 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.111555 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.111532 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.113882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.113848 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "kube-api-access-8btj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:33.115151 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.115094 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.117011 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.116817 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs" pod "42f903d4-b68e-4291-869f-2092f360a459" (UID: "42f903d4-b68e-4291-869f-2092f360a459"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:33.212885 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.212850 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f903d4-b68e-4291-869f-2092f360a459-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.213143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.213119 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btj6\" (UniqueName: \"kubernetes.io/projected/42f903d4-b68e-4291-869f-2092f360a459-kube-api-access-8btj6\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.213257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.213147 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f903d4-b68e-4291-869f-2092f360a459-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.457571 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.457535 5136 generic.go:356] "Generic (PLEG): container finished" podID="42f903d4-b68e-4291-869f-2092f360a459" containerID="860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387" exitCode=0 Nov 02 21:13:33.457817 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.457589 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" event={"ID":"42f903d4-b68e-4291-869f-2092f360a459","Type":"ContainerDied","Data":"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387"} Nov 02 21:13:33.457817 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.457661 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" event={"ID":"42f903d4-b68e-4291-869f-2092f360a459","Type":"ContainerDied","Data":"41563519e6da740c0d957d8bd9ae6ea98b8cd617c8ce8e9bbebb992d28f5583f"} Nov 02 21:13:33.457817 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.457684 5136 scope.go:122] "RemoveContainer" containerID="860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387" Nov 02 21:13:33.457817 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.457617 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q" Nov 02 21:13:33.468848 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.468812 5136 scope.go:122] "RemoveContainer" containerID="860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387" Nov 02 21:13:33.469229 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:13:33.469104 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387\": container with ID starting with 860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387 not found: ID does not exist" containerID="860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387" Nov 02 21:13:33.469229 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.469139 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387"} err="failed to get container status \"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387\": rpc error: code = NotFound desc = could not find container \"860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387\": container with ID starting with 860ada4dca6a8083a117ca6f99d05bf3de6a856eb045bd15c5051a7303e7b387 not found: ID does not exist" Nov 02 21:13:33.613434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.613403 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 21:13:33.661682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:33.661653 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/premium-w4-c25349-kserve-66f686f5b4-b5h7q"] Nov 02 21:13:34.547480 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:34.547451 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f903d4-b68e-4291-869f-2092f360a459" path="/var/lib/kubelet/pods/42f903d4-b68e-4291-869f-2092f360a459/volumes" Nov 02 21:13:36.538460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:36.538426 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:36.552414 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:36.552389 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:37.491394 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:37.491360 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:13:49.058310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.058271 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 21:13:49.059079 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.059036 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" podUID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" containerName="maas-api" containerID="cri-o://f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665" gracePeriod=30 Nov 02 21:13:49.333646 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.333619 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 21:13:49.496487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.496438 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") pod \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " Nov 02 21:13:49.496487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.496489 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\") pod \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " Nov 02 21:13:49.496794 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.496557 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\") pod \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " Nov 02 21:13:49.496794 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.496621 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\" (UniqueName: \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\") pod \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\" (UID: \"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f\") " Nov 02 21:13:49.496938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.496892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca" pod "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" (UID: "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:49.497120 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.497097 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:49.499322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.499291 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls" pod "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" (UID: "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:49.499495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.499462 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9" pod "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" (UID: "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f"). InnerVolumeSpecName "kube-api-access-xczn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:49.499614 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.499495 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs" pod "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" (UID: "f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:49.550051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.550019 5136 generic.go:356] "Generic (PLEG): container finished" podID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" containerID="f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665" exitCode=0 Nov 02 21:13:49.550209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.550072 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" Nov 02 21:13:49.550209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.550097 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" event={"ID":"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f","Type":"ContainerDied","Data":"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665"} Nov 02 21:13:49.550209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.550154 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv" event={"ID":"f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f","Type":"ContainerDied","Data":"8f1cf9631b6e8a0d9d999afc9a0420b65576e383080996d140e48eed78948012"} Nov 02 21:13:49.550209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.550180 5136 scope.go:122] "RemoveContainer" containerID="f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665" Nov 02 21:13:49.561782 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.561723 5136 scope.go:122] "RemoveContainer" containerID="f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665" Nov 02 21:13:49.565209 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:13:49.565073 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665\": container with ID starting with f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665 not found: ID does not exist" containerID="f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665" Nov 02 21:13:49.565209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.565129 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665"} err="failed to get container status \"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665\": rpc error: code = NotFound desc = could not find container \"f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665\": container with ID starting with f3e12ae84330570a61b4f26906a41854f202134ef51de9ded88fb4bde7658665 not found: ID does not exist" Nov 02 21:13:49.598597 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.598559 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xczn9\" (UniqueName: \"kubernetes.io/projected/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-kube-api-access-xczn9\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:49.598597 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.598594 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:49.598847 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.598609 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:49.705688 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.705659 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 21:13:49.709655 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:49.709628 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c25349-74ddcd99ff-mjztv"] Nov 02 21:13:50.546645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:50.546615 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" path="/var/lib/kubelet/pods/f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f/volumes" Nov 02 21:13:51.003607 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:13:51.003570 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:13:58.208795 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:58.208763 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 21:13:58.209263 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:58.208968 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" containerID="cri-o://01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f" gracePeriod=30 Nov 02 21:13:59.697434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:59.697398 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 21:13:59.697925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:13:59.697617 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" containerID="cri-o://ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c" gracePeriod=30 Nov 02 21:14:03.467530 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.467501 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 21:14:03.563085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563020 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563055 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563084 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563126 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563149 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563270 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563279 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:03.563341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563318 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563394 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563457 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\") pod \"da8adf00-4cf7-4663-ae94-8f70534eb244\" (UID: \"da8adf00-4cf7-4663-ae94-8f70534eb244\") " Nov 02 21:14:03.563624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563496 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:03.563624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563541 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:03.563624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563578 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563652 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563874 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563895 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563910 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563922 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8adf00-4cf7-4663-ae94-8f70534eb244-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.563958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.563937 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.565673 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.565649 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "kube-api-access-blt2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:03.565794 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.565719 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:14:03.565844 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.565816 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:03.565880 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.565839 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token" pod "da8adf00-4cf7-4663-ae94-8f70534eb244" (UID: "da8adf00-4cf7-4663-ae94-8f70534eb244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:03.633441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.633403 5136 generic.go:356] "Generic (PLEG): container finished" podID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerID="01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f" exitCode=0 Nov 02 21:14:03.633608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.633454 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" Nov 02 21:14:03.633608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.633591 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" event={"ID":"da8adf00-4cf7-4663-ae94-8f70534eb244","Type":"ContainerDied","Data":"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f"} Nov 02 21:14:03.633720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.633641 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs" event={"ID":"da8adf00-4cf7-4663-ae94-8f70534eb244","Type":"ContainerDied","Data":"07cc85b41c76932cf13f7585a3f790d29239b27065b2e93e7a1a0742743afdc9"} Nov 02 21:14:03.633720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.633670 5136 scope.go:122] "RemoveContainer" containerID="01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f" Nov 02 21:14:03.649344 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.649323 5136 scope.go:122] "RemoveContainer" containerID="01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f" Nov 02 21:14:03.649608 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:03.649590 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f\": container with ID starting with 01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f not found: ID does not exist" containerID="01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f" Nov 02 21:14:03.649679 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.649621 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f"} err="failed to get container status \"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f\": rpc error: code = NotFound desc = could not find container \"01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f\": container with ID starting with 01c89e32a4f59a65f90a9d5675c8a3e4f265202368edcea8519158f9af81531f not found: ID does not exist" Nov 02 21:14:03.665287 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.665265 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.665287 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.665287 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blt2d\" (UniqueName: \"kubernetes.io/projected/da8adf00-4cf7-4663-ae94-8f70534eb244-kube-api-access-blt2d\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.665396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.665296 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.665396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.665305 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8adf00-4cf7-4663-ae94-8f70534eb244-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:03.789060 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.789027 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 21:14:03.793383 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:03.793348 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-k7rzs"] Nov 02 21:14:04.547041 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:04.547007 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" path="/var/lib/kubelet/pods/da8adf00-4cf7-4663-ae94-8f70534eb244/volumes" Nov 02 21:14:04.968182 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:04.968158 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 21:14:05.079863 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.079777 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.079861 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.079888 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.079972 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080005 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080046 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080084 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080146 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080176 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\") pod \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\" (UID: \"0a9619b5-758e-4ae0-a88d-d0ea4cd17988\") " Nov 02 21:14:05.080286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080198 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:05.080759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.080719 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.081172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.081145 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:05.081300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.081236 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:05.081551 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.081377 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:05.081915 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.081851 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:05.083137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.083088 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:05.084078 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.084042 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:14:05.084489 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.084462 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "kube-api-access-gsl5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:05.084579 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.084488 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token" pod "0a9619b5-758e-4ae0-a88d-d0ea4cd17988" (UID: "0a9619b5-758e-4ae0-a88d-d0ea4cd17988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:05.181616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181586 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181612 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181616 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181622 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181632 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181641 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsl5j\" (UniqueName: \"kubernetes.io/projected/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-kube-api-access-gsl5j\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181650 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181659 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.181936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.181668 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9619b5-758e-4ae0-a88d-d0ea4cd17988-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:05.646696 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.646666 5136 generic.go:356] "Generic (PLEG): container finished" podID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerID="ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c" exitCode=0 Nov 02 21:14:05.647130 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.646713 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" Nov 02 21:14:05.647130 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.646764 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerDied","Data":"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c"} Nov 02 21:14:05.647130 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.646809 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j" event={"ID":"0a9619b5-758e-4ae0-a88d-d0ea4cd17988","Type":"ContainerDied","Data":"629dc8ebc47d3edef41fbd5a0b8019146a81a05cfb9ce8dc87a807af57f983d8"} Nov 02 21:14:05.647130 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.646835 5136 scope.go:122] "RemoveContainer" containerID="ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c" Nov 02 21:14:05.665423 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.665399 5136 scope.go:122] "RemoveContainer" containerID="f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843" Nov 02 21:14:05.674205 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.674179 5136 scope.go:122] "RemoveContainer" containerID="ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c" Nov 02 21:14:05.674468 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:05.674448 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c\": container with ID starting with ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c not found: ID does not exist" containerID="ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c" Nov 02 21:14:05.674547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.674480 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c"} err="failed to get container status \"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c\": rpc error: code = NotFound desc = could not find container \"ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c\": container with ID starting with ee221208c2786ca2baa7e99a84e1a1f2818a30c08c91756214d0cd8dc7bab55c not found: ID does not exist" Nov 02 21:14:05.674547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.674505 5136 scope.go:122] "RemoveContainer" containerID="f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843" Nov 02 21:14:05.674804 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:05.674784 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843\": container with ID starting with f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843 not found: ID does not exist" containerID="f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843" Nov 02 21:14:05.674859 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.674810 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843"} err="failed to get container status \"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843\": rpc error: code = NotFound desc = could not find container \"f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843\": container with ID starting with f7bc27295daaf339d1288db8a95c193c083498e8cb4f0f2b4612863fbb18d843 not found: ID does not exist" Nov 02 21:14:05.802230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.802199 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 21:14:05.849660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:05.849631 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-44a484-openshift-default-94cf8b987-m2j5j"] Nov 02 21:14:06.547319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:06.547287 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" path="/var/lib/kubelet/pods/0a9619b5-758e-4ae0-a88d-d0ea4cd17988/volumes" Nov 02 21:14:21.004024 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:21.003939 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:14:26.524920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.524885 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 21:14:26.525400 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.525077 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" podUID="1dab15d7-86a1-45a7-804c-348e82f90833" containerName="maas-api" containerID="cri-o://b090f78e78012c33da11aaadcf8db5263269c1a17523e65ee254aeb661798dba" gracePeriod=30 Nov 02 21:14:26.762901 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.762860 5136 generic.go:356] "Generic (PLEG): container finished" podID="1dab15d7-86a1-45a7-804c-348e82f90833" containerID="b090f78e78012c33da11aaadcf8db5263269c1a17523e65ee254aeb661798dba" exitCode=0 Nov 02 21:14:26.763069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.762925 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" event={"ID":"1dab15d7-86a1-45a7-804c-348e82f90833","Type":"ContainerDied","Data":"b090f78e78012c33da11aaadcf8db5263269c1a17523e65ee254aeb661798dba"} Nov 02 21:14:26.792209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.792151 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 21:14:26.809155 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809127 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\") pod \"1dab15d7-86a1-45a7-804c-348e82f90833\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " Nov 02 21:14:26.809303 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809216 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\" (UniqueName: \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\") pod \"1dab15d7-86a1-45a7-804c-348e82f90833\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " Nov 02 21:14:26.809303 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809248 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\") pod \"1dab15d7-86a1-45a7-804c-348e82f90833\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " Nov 02 21:14:26.809434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809317 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") pod \"1dab15d7-86a1-45a7-804c-348e82f90833\" (UID: \"1dab15d7-86a1-45a7-804c-348e82f90833\") " Nov 02 21:14:26.809555 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809531 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca" pod "1dab15d7-86a1-45a7-804c-348e82f90833" (UID: "1dab15d7-86a1-45a7-804c-348e82f90833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:26.809713 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.809696 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab15d7-86a1-45a7-804c-348e82f90833-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.811403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.811382 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs" pod "1dab15d7-86a1-45a7-804c-348e82f90833" (UID: "1dab15d7-86a1-45a7-804c-348e82f90833"). InnerVolumeSpecName "kube-api-access-dsxqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:26.811499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.811425 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs" pod "1dab15d7-86a1-45a7-804c-348e82f90833" (UID: "1dab15d7-86a1-45a7-804c-348e82f90833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:26.811499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.811445 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls" pod "1dab15d7-86a1-45a7-804c-348e82f90833" (UID: "1dab15d7-86a1-45a7-804c-348e82f90833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:26.914429 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.914137 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.914429 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.914184 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxqs\" (UniqueName: \"kubernetes.io/projected/1dab15d7-86a1-45a7-804c-348e82f90833-kube-api-access-dsxqs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.914429 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:26.914200 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab15d7-86a1-45a7-804c-348e82f90833-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:27.770578 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:27.770543 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" event={"ID":"1dab15d7-86a1-45a7-804c-348e82f90833","Type":"ContainerDied","Data":"d8f06648876fd7bea236b7dcf6d1beb51b3e7dbaab12a3780b0dda1e4e3ef34e"} Nov 02 21:14:27.771028 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:27.770610 5136 scope.go:122] "RemoveContainer" containerID="b090f78e78012c33da11aaadcf8db5263269c1a17523e65ee254aeb661798dba" Nov 02 21:14:27.771028 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:27.770558 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t" Nov 02 21:14:27.925869 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:27.925826 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 21:14:27.972986 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:27.972952 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-09ecd0-5f89d7ddc-5dr4t"] Nov 02 21:14:28.546228 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:28.546195 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dab15d7-86a1-45a7-804c-348e82f90833" path="/var/lib/kubelet/pods/1dab15d7-86a1-45a7-804c-348e82f90833/volumes" Nov 02 21:14:42.418404 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:42.418368 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 21:14:51.003724 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:51.003676 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:14:51.222939 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:51.222904 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 21:14:56.224155 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:56.224115 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" containerID="cri-o://2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075" gracePeriod=25 Nov 02 21:14:57.036722 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.036687 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:14:57.438968 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.438878 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" containerID="cri-o://145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955" gracePeriod=45 Nov 02 21:14:57.456793 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.456764 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused" Nov 02 21:14:57.682553 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.682527 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 21:14:57.830896 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.830867 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831100 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.830953 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831100 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.830984 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\" (UniqueName: \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831100 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831078 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831112 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831145 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\") pod \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\" (UID: \"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61\") " Nov 02 21:14:57.831291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831180 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.831291 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831261 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.831593 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831562 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.831593 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831568 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.832084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.831618 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.833638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.833605 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "kube-api-access-b577m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:57.833807 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.833757 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.833807 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.833780 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs" pod "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" (UID: "8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:57.932670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.932635 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b577m\" (UniqueName: \"kubernetes.io/projected/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-kube-api-access-b577m\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.932670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.932664 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.932670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.932674 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.932988 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.932682 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.959759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.959696 5136 generic.go:356] "Generic (PLEG): container finished" podID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerID="145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955" exitCode=0 Nov 02 21:14:57.959958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.959775 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" Nov 02 21:14:57.959958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.959794 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" event={"ID":"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61","Type":"ContainerDied","Data":"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955"} Nov 02 21:14:57.959958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.959853 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs" event={"ID":"8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61","Type":"ContainerDied","Data":"84723b8f55eb4b0a97858adb4321bdc9265c5594472c4cd55ba2bfd6b254470a"} Nov 02 21:14:57.959958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.959878 5136 scope.go:122] "RemoveContainer" containerID="145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955" Nov 02 21:14:57.973133 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.973112 5136 scope.go:122] "RemoveContainer" containerID="145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955" Nov 02 21:14:57.973434 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:14:57.973416 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955\": container with ID starting with 145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955 not found: ID does not exist" containerID="145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955" Nov 02 21:14:57.973498 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:57.973445 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955"} err="failed to get container status \"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955\": rpc error: code = NotFound desc = could not find container \"145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955\": container with ID starting with 145107aac71738a4d4a0ebc60f9d150a7b000f924e552dd85494045d41d67955 not found: ID does not exist" Nov 02 21:14:58.116882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:58.116804 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 21:14:58.120687 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:58.120658 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-2-w1-bbee67-kserve-7c589d5cf8-f6drs"] Nov 02 21:14:58.547122 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:14:58.547090 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" path="/var/lib/kubelet/pods/8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61/volumes" Nov 02 21:15:07.037210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:07.037168 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:15:16.487064 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.487006 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 21:15:16.533285 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.533258 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\" (UniqueName: \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\") pod \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " Nov 02 21:15:16.533462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.533448 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\") pod \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\" (UID: \"7b6f95a6-eee7-4b3e-8aff-fb99a6214377\") " Nov 02 21:15:16.533888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.533861 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume" pod "7b6f95a6-eee7-4b3e-8aff-fb99a6214377" (UID: "7b6f95a6-eee7-4b3e-8aff-fb99a6214377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:16.535346 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.535319 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5" pod "7b6f95a6-eee7-4b3e-8aff-fb99a6214377" (UID: "7b6f95a6-eee7-4b3e-8aff-fb99a6214377"). InnerVolumeSpecName "kube-api-access-tz4x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:16.634970 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.634939 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:16.634970 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:16.634966 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4x5\" (UniqueName: \"kubernetes.io/projected/7b6f95a6-eee7-4b3e-8aff-fb99a6214377-kube-api-access-tz4x5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:17.059251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.059211 5136 generic.go:356] "Generic (PLEG): container finished" podID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerID="2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075" exitCode=0 Nov 02 21:15:17.059471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.059259 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" Nov 02 21:15:17.059471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.059298 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" event={"ID":"7b6f95a6-eee7-4b3e-8aff-fb99a6214377","Type":"ContainerDied","Data":"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075"} Nov 02 21:15:17.059471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.059366 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4" event={"ID":"7b6f95a6-eee7-4b3e-8aff-fb99a6214377","Type":"ContainerDied","Data":"8602d5927bb2484edaff185c64dc7eeaafcf5bcd712f6f967c55125699ac4deb"} Nov 02 21:15:17.059471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.059396 5136 scope.go:122] "RemoveContainer" containerID="2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075" Nov 02 21:15:17.068543 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.068524 5136 scope.go:122] "RemoveContainer" containerID="2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075" Nov 02 21:15:17.068865 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:17.068839 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075\": container with ID starting with 2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075 not found: ID does not exist" containerID="2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075" Nov 02 21:15:17.068934 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.068876 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075"} err="failed to get container status \"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075\": rpc error: code = NotFound desc = could not find container \"2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075\": container with ID starting with 2b40dd3f76c5e99af2f7bd60d3918b228a22c6fc2853d1a4c8f0d8c55c832075 not found: ID does not exist" Nov 02 21:15:17.215713 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.215682 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 21:15:17.262972 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:17.262943 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bbee67-64b498bb7f-z5js4"] Nov 02 21:15:18.545674 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:18.545641 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" path="/var/lib/kubelet/pods/7b6f95a6-eee7-4b3e-8aff-fb99a6214377/volumes" Nov 02 21:15:21.004128 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:21.004087 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:15:22.841689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:22.841656 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 21:15:22.842109 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:22.841875 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" podUID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" containerName="maas-api" containerID="cri-o://9ccb45fb91c7a3a01c3f8d820e077bcf109fb86531e214a192717c8da0c2ec7d" gracePeriod=30 Nov 02 21:15:23.095697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.095628 5136 generic.go:356] "Generic (PLEG): container finished" podID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" containerID="9ccb45fb91c7a3a01c3f8d820e077bcf109fb86531e214a192717c8da0c2ec7d" exitCode=0 Nov 02 21:15:23.095697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.095679 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" event={"ID":"1d758097-e052-4a9e-b5ee-4ee30c33b38f","Type":"ContainerDied","Data":"9ccb45fb91c7a3a01c3f8d820e077bcf109fb86531e214a192717c8da0c2ec7d"} Nov 02 21:15:23.095868 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.095750 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" event={"ID":"1d758097-e052-4a9e-b5ee-4ee30c33b38f","Type":"ContainerDied","Data":"119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102"} Nov 02 21:15:23.095868 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.095773 5136 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119741a33999fc4007e1874ef7e36c08cff062ebd3ea7ab8fe701ed63ec76102" Nov 02 21:15:23.107319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.107300 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 21:15:23.204954 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.204924 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") pod \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " Nov 02 21:15:23.205137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.204987 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\" (UniqueName: \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\") pod \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " Nov 02 21:15:23.205137 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.205050 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\") pod \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " Nov 02 21:15:23.205263 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.205240 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\") pod \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\" (UID: \"1d758097-e052-4a9e-b5ee-4ee30c33b38f\") " Nov 02 21:15:23.205467 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.205442 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca" pod "1d758097-e052-4a9e-b5ee-4ee30c33b38f" (UID: "1d758097-e052-4a9e-b5ee-4ee30c33b38f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:23.207207 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.207176 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls" pod "1d758097-e052-4a9e-b5ee-4ee30c33b38f" (UID: "1d758097-e052-4a9e-b5ee-4ee30c33b38f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:23.207331 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.207220 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d" pod "1d758097-e052-4a9e-b5ee-4ee30c33b38f" (UID: "1d758097-e052-4a9e-b5ee-4ee30c33b38f"). InnerVolumeSpecName "kube-api-access-qq54d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:23.207331 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.207276 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs" pod "1d758097-e052-4a9e-b5ee-4ee30c33b38f" (UID: "1d758097-e052-4a9e-b5ee-4ee30c33b38f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:23.306166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.306131 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:23.306166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.306159 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d758097-e052-4a9e-b5ee-4ee30c33b38f-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:23.306166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.306171 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq54d\" (UniqueName: \"kubernetes.io/projected/1d758097-e052-4a9e-b5ee-4ee30c33b38f-kube-api-access-qq54d\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:23.306427 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:23.306181 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d758097-e052-4a9e-b5ee-4ee30c33b38f-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:24.101866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:24.101832 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7" Nov 02 21:15:24.256121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:24.256083 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 21:15:24.303388 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:24.303356 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bbee67-598444c945-nxcp7"] Nov 02 21:15:24.545444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:24.545407 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" path="/var/lib/kubelet/pods/1d758097-e052-4a9e-b5ee-4ee30c33b38f/volumes" Nov 02 21:15:27.403376 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:27.403344 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 21:15:27.403804 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:27.403548 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" containerID="cri-o://2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f" gracePeriod=30 Nov 02 21:15:31.491792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:31.491746 5136 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:15:31.492224 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:31.491819 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:15:32.659451 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.659429 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 21:15:32.805823 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.805796 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.805867 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.805912 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.805959 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.805992 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806030 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806077 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806128 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806222 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\") pod \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\" (UID: \"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01\") " Nov 02 21:15:32.806265 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806243 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:32.806526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806257 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:32.806526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806323 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:32.806710 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806599 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.806710 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806618 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.806710 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806631 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.806710 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.806646 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:32.807090 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.807068 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:32.808224 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.808197 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:32.808338 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.808262 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:15:32.808454 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.808434 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "kube-api-access-bnq9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:32.808549 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.808527 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy" pod "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" (UID: "abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:32.907853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907821 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq9q\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-kube-api-access-bnq9q\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.907853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907848 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.907853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907858 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.908108 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907867 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.908108 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907876 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:32.908108 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:32.907886 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:33.154686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.154587 5136 generic.go:356] "Generic (PLEG): container finished" podID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerID="2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f" exitCode=0 Nov 02 21:15:33.154686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.154632 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" Nov 02 21:15:33.154686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.154655 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerDied","Data":"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f"} Nov 02 21:15:33.155021 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.154723 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h" event={"ID":"abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01","Type":"ContainerDied","Data":"a1d76b801ee70fc81d59db03ce0c990af0089b5ecb9f5db91fbaab38ee66c444"} Nov 02 21:15:33.155021 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.154755 5136 scope.go:122] "RemoveContainer" containerID="2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f" Nov 02 21:15:33.165998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.165974 5136 scope.go:122] "RemoveContainer" containerID="b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457" Nov 02 21:15:33.177297 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.177270 5136 scope.go:122] "RemoveContainer" containerID="2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f" Nov 02 21:15:33.177595 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:33.177575 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f\": container with ID starting with 2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f not found: ID does not exist" containerID="2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f" Nov 02 21:15:33.177648 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.177603 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f"} err="failed to get container status \"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f\": rpc error: code = NotFound desc = could not find container \"2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f\": container with ID starting with 2f6828306ea1158ebd0f06f81d496f67dc1527dff133ab03871d646c35d4145f not found: ID does not exist" Nov 02 21:15:33.177648 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.177621 5136 scope.go:122] "RemoveContainer" containerID="b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457" Nov 02 21:15:33.177932 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:33.177914 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457\": container with ID starting with b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457 not found: ID does not exist" containerID="b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457" Nov 02 21:15:33.178005 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.177934 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457"} err="failed to get container status \"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457\": rpc error: code = NotFound desc = could not find container \"b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457\": container with ID starting with b8e77ef3126fecf088fbfcdd3ba1e09ecd8656ce76a6ce0af870c0b467d0b457 not found: ID does not exist" Nov 02 21:15:33.312090 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.312044 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 21:15:33.315338 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:33.315314 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-dhd6h"] Nov 02 21:15:34.545520 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:34.545488 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" path="/var/lib/kubelet/pods/abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01/volumes" Nov 02 21:15:37.243018 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:37.242988 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 21:15:37.243469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:37.243193 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" containerID="cri-o://71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f" gracePeriod=30 Nov 02 21:15:37.404531 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:37.404493 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-4878k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:15:37.404702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:37.404545 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:15:38.348128 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:38.348089 5136 scope.go:122] "RemoveContainer" containerID="9ccb45fb91c7a3a01c3f8d820e077bcf109fb86531e214a192717c8da0c2ec7d" Nov 02 21:15:42.497005 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.496979 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 21:15:42.606840 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606772 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.606840 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606829 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606848 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606872 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606891 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.606930 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607015 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607068 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607109 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\") pod \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\" (UID: \"765c1df4-1f03-4847-b5a6-f4a075c40ee8\") " Nov 02 21:15:42.607315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607191 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:42.607492 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607364 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:42.607492 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607478 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:42.607605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607557 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:42.607605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607562 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.607605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607585 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:42.607605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.607599 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.609066 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.609045 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:15:42.609264 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.609249 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "kube-api-access-l8972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:42.609518 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.609492 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:42.609619 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.609523 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token" pod "765c1df4-1f03-4847-b5a6-f4a075c40ee8" (UID: "765c1df4-1f03-4847-b5a6-f4a075c40ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:42.709004 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.708972 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8972\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-kube-api-access-l8972\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709004 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709001 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709012 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709021 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709031 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709039 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:42.709192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:42.709047 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765c1df4-1f03-4847-b5a6-f4a075c40ee8-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:43.207129 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.207094 5136 generic.go:356] "Generic (PLEG): container finished" podID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerID="71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f" exitCode=0 Nov 02 21:15:43.207309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.207133 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" Nov 02 21:15:43.207309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.207173 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" event={"ID":"765c1df4-1f03-4847-b5a6-f4a075c40ee8","Type":"ContainerDied","Data":"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f"} Nov 02 21:15:43.207309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.207229 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k" event={"ID":"765c1df4-1f03-4847-b5a6-f4a075c40ee8","Type":"ContainerDied","Data":"f99ea114b87abe9936eb1dbe19606581798056f51f1cc07cd618ef2904a9d635"} Nov 02 21:15:43.207309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.207251 5136 scope.go:122] "RemoveContainer" containerID="71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f" Nov 02 21:15:43.217630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.217609 5136 scope.go:122] "RemoveContainer" containerID="71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f" Nov 02 21:15:43.217988 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:43.217967 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f\": container with ID starting with 71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f not found: ID does not exist" containerID="71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f" Nov 02 21:15:43.218083 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.217995 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f"} err="failed to get container status \"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f\": rpc error: code = NotFound desc = could not find container \"71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f\": container with ID starting with 71da6ca69bd45da110484f4b7dcd3cc8bc411f6636bf515b5a73a0b754f5422f not found: ID does not exist" Nov 02 21:15:43.410938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.410904 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 21:15:43.458652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:43.458575 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-4878k"] Nov 02 21:15:44.545271 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:15:44.545238 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" path="/var/lib/kubelet/pods/765c1df4-1f03-4847-b5a6-f4a075c40ee8/volumes" Nov 02 21:15:51.003722 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:15:51.003629 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:16:15.896373 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:15.896338 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:16:20.810757 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:20.810700 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:16:21.003639 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:16:21.003606 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:16:30.917434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:30.917392 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" podUID="9be482bd-db18-4449-b943-8c2d93236048" containerName="main" containerID="cri-o://a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e" gracePeriod=45 Nov 02 21:16:30.924068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:30.924035 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" podUID="9be482bd-db18-4449-b943-8c2d93236048" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": dial tcp 10.133.0.201:8000: connect: connection refused" Nov 02 21:16:31.182358 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.182299 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:16:31.283925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.283896 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.283968 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\" (UniqueName: \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284009 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284035 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\" (UniqueName: \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284287 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284113 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284287 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284140 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\") pod \"9be482bd-db18-4449-b943-8c2d93236048\" (UID: \"9be482bd-db18-4449-b943-8c2d93236048\") " Nov 02 21:16:31.284406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284304 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:31.284533 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284387 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:31.284533 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284458 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:31.284699 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284673 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.284699 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284700 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.284923 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.284716 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.286193 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.286141 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:31.286193 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.286148 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:31.286318 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.286242 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b" pod "9be482bd-db18-4449-b943-8c2d93236048" (UID: "9be482bd-db18-4449-b943-8c2d93236048"). InnerVolumeSpecName "kube-api-access-kxb5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:31.386083 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.386050 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be482bd-db18-4449-b943-8c2d93236048-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.386083 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.386078 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be482bd-db18-4449-b943-8c2d93236048-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.386268 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.386106 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxb5b\" (UniqueName: \"kubernetes.io/projected/9be482bd-db18-4449-b943-8c2d93236048-kube-api-access-kxb5b\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:31.448465 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.448433 5136 generic.go:356] "Generic (PLEG): container finished" podID="9be482bd-db18-4449-b943-8c2d93236048" containerID="a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e" exitCode=0 Nov 02 21:16:31.448638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.448491 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" Nov 02 21:16:31.448638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.448534 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" event={"ID":"9be482bd-db18-4449-b943-8c2d93236048","Type":"ContainerDied","Data":"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e"} Nov 02 21:16:31.448638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.448581 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n" event={"ID":"9be482bd-db18-4449-b943-8c2d93236048","Type":"ContainerDied","Data":"045d0e119b306f285008516c25c51ec3878c4661d0c3acce420e68053e3c460d"} Nov 02 21:16:31.448638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.448603 5136 scope.go:122] "RemoveContainer" containerID="a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e" Nov 02 21:16:31.458491 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.458472 5136 scope.go:122] "RemoveContainer" containerID="a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e" Nov 02 21:16:31.458753 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:16:31.458715 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e\": container with ID starting with a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e not found: ID does not exist" containerID="a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e" Nov 02 21:16:31.458811 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.458763 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e"} err="failed to get container status \"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e\": rpc error: code = NotFound desc = could not find container \"a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e\": container with ID starting with a14d00c92bd49f6aec811f0064984ede33960558145877a425a897917f22ef4e not found: ID does not exist" Nov 02 21:16:31.602556 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.602526 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:16:31.605875 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:31.605837 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-76c687/sub-test-model-76c687-kserve-7b5bdf459-xjc6n"] Nov 02 21:16:32.545789 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:32.545752 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be482bd-db18-4449-b943-8c2d93236048" path="/var/lib/kubelet/pods/9be482bd-db18-4449-b943-8c2d93236048/volumes" Nov 02 21:16:35.830911 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:35.830874 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" podUID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" containerName="main" containerID="cri-o://808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04" gracePeriod=45 Nov 02 21:16:36.089720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.089666 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:16:36.234649 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234619 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.234845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234659 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\" (UniqueName: \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.234845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234701 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.234845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234748 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.234845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234829 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.235145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234864 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\") pod \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\" (UID: \"ac130cd9-f3ab-4b01-9e38-7d7647f62468\") " Nov 02 21:16:36.235145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234971 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:36.235145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.234986 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:36.235315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.235161 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:36.235315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.235282 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.235315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.235302 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.235435 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.235316 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.237024 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.236993 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:36.237024 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.236999 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:36.237258 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.237243 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj" pod "ac130cd9-f3ab-4b01-9e38-7d7647f62468" (UID: "ac130cd9-f3ab-4b01-9e38-7d7647f62468"). InnerVolumeSpecName "kube-api-access-chcwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:36.336317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.336288 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac130cd9-f3ab-4b01-9e38-7d7647f62468-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.336317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.336312 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcwj\" (UniqueName: \"kubernetes.io/projected/ac130cd9-f3ab-4b01-9e38-7d7647f62468-kube-api-access-chcwj\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.336317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.336323 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac130cd9-f3ab-4b01-9e38-7d7647f62468-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.475014 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.474978 5136 generic.go:356] "Generic (PLEG): container finished" podID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" containerID="808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04" exitCode=0 Nov 02 21:16:36.475194 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.475022 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" event={"ID":"ac130cd9-f3ab-4b01-9e38-7d7647f62468","Type":"ContainerDied","Data":"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04"} Nov 02 21:16:36.475194 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.475048 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" Nov 02 21:16:36.475194 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.475069 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn" event={"ID":"ac130cd9-f3ab-4b01-9e38-7d7647f62468","Type":"ContainerDied","Data":"be92f0a4f303c8b5d566b306fa33ba9ad94f4500833735ec4d150df1ed096d0f"} Nov 02 21:16:36.475194 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.475090 5136 scope.go:122] "RemoveContainer" containerID="808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04" Nov 02 21:16:36.484104 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.484083 5136 scope.go:122] "RemoveContainer" containerID="808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04" Nov 02 21:16:36.484375 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:16:36.484354 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04\": container with ID starting with 808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04 not found: ID does not exist" containerID="808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04" Nov 02 21:16:36.484443 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.484387 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04"} err="failed to get container status \"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04\": rpc error: code = NotFound desc = could not find container \"808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04\": container with ID starting with 808ef2b70a095202f8c5f0764e9cf48a12ec58ea65aa76f8777a34c5e9699e04 not found: ID does not exist" Nov 02 21:16:36.629192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.629163 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:16:36.631950 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:36.631923 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6b12ae/sub-test-model-6b12ae-kserve-5bdfdfbb55-zk8kn"] Nov 02 21:16:38.546193 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:16:38.546162 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" path="/var/lib/kubelet/pods/ac130cd9-f3ab-4b01-9e38-7d7647f62468/volumes" Nov 02 21:16:51.003272 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:16:51.003231 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:17:04.002971 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:04.002930 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 21:17:09.003657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:09.003589 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" containerID="cri-o://9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2" gracePeriod=25 Nov 02 21:17:13.770329 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:13.770287 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:17.865355 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:17.865271 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 21:17:17.865794 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:17.865459 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" podUID="34e506f6-598b-480f-876e-e7cbb08ab632" containerName="maas-api" containerID="cri-o://45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b" gracePeriod=30 Nov 02 21:17:18.116263 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.116199 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 21:17:18.250245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250215 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\") pod \"34e506f6-598b-480f-876e-e7cbb08ab632\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " Nov 02 21:17:18.250412 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250327 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\") pod \"34e506f6-598b-480f-876e-e7cbb08ab632\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " Nov 02 21:17:18.250462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250418 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\" (UniqueName: \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\") pod \"34e506f6-598b-480f-876e-e7cbb08ab632\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " Nov 02 21:17:18.250500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250467 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") pod \"34e506f6-598b-480f-876e-e7cbb08ab632\" (UID: \"34e506f6-598b-480f-876e-e7cbb08ab632\") " Nov 02 21:17:18.250697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250676 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca" pod "34e506f6-598b-480f-876e-e7cbb08ab632" (UID: "34e506f6-598b-480f-876e-e7cbb08ab632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:18.250970 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.250946 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e506f6-598b-480f-876e-e7cbb08ab632-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:18.252575 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.252547 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs" pod "34e506f6-598b-480f-876e-e7cbb08ab632" (UID: "34e506f6-598b-480f-876e-e7cbb08ab632"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:18.252677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.252581 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls" pod "34e506f6-598b-480f-876e-e7cbb08ab632" (UID: "34e506f6-598b-480f-876e-e7cbb08ab632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:18.252677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.252606 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl" pod "34e506f6-598b-480f-876e-e7cbb08ab632" (UID: "34e506f6-598b-480f-876e-e7cbb08ab632"). InnerVolumeSpecName "kube-api-access-hqrbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:18.352205 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.352169 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrbl\" (UniqueName: \"kubernetes.io/projected/34e506f6-598b-480f-876e-e7cbb08ab632-kube-api-access-hqrbl\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:18.352205 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.352202 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:18.352410 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.352220 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e506f6-598b-480f-876e-e7cbb08ab632-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:18.678602 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.678569 5136 generic.go:356] "Generic (PLEG): container finished" podID="34e506f6-598b-480f-876e-e7cbb08ab632" containerID="45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b" exitCode=0 Nov 02 21:17:18.678787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.678629 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" Nov 02 21:17:18.678787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.678672 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" event={"ID":"34e506f6-598b-480f-876e-e7cbb08ab632","Type":"ContainerDied","Data":"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b"} Nov 02 21:17:18.678787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.678772 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5" event={"ID":"34e506f6-598b-480f-876e-e7cbb08ab632","Type":"ContainerDied","Data":"b7c0bfd43ef40b8e083a4d8b95d19f594edef625c47c5a781d8afb23db3d13f1"} Nov 02 21:17:18.678953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.678797 5136 scope.go:122] "RemoveContainer" containerID="45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b" Nov 02 21:17:18.689959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.689937 5136 scope.go:122] "RemoveContainer" containerID="45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b" Nov 02 21:17:18.690300 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:18.690276 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b\": container with ID starting with 45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b not found: ID does not exist" containerID="45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b" Nov 02 21:17:18.690373 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.690306 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b"} err="failed to get container status \"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b\": rpc error: code = NotFound desc = could not find container \"45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b\": container with ID starting with 45ffa1a7b34ed237323e066d8859f7f53983f6c7a9ffb088e163bb9ec534762b not found: ID does not exist" Nov 02 21:17:18.833640 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.833563 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 21:17:18.881015 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:18.880981 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-76c687-695cb797f7-f2vb5"] Nov 02 21:17:20.546182 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:20.546142 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e506f6-598b-480f-876e-e7cbb08ab632" path="/var/lib/kubelet/pods/34e506f6-598b-480f-876e-e7cbb08ab632/volumes" Nov 02 21:17:21.003505 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:21.003470 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:17:23.769997 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:23.769917 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:29.257986 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.257960 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 21:17:29.365335 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.365223 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\") pod \"82af6c97-0661-4765-b9ee-b3928848fece\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " Nov 02 21:17:29.365335 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.365303 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\" (UniqueName: \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\") pod \"82af6c97-0661-4765-b9ee-b3928848fece\" (UID: \"82af6c97-0661-4765-b9ee-b3928848fece\") " Nov 02 21:17:29.365970 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.365934 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume" pod "82af6c97-0661-4765-b9ee-b3928848fece" (UID: "82af6c97-0661-4765-b9ee-b3928848fece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:29.368473 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.368437 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4" pod "82af6c97-0661-4765-b9ee-b3928848fece" (UID: "82af6c97-0661-4765-b9ee-b3928848fece"). InnerVolumeSpecName "kube-api-access-lmnv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:29.467097 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.467069 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af6c97-0661-4765-b9ee-b3928848fece-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:29.467097 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.467095 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnv4\" (UniqueName: \"kubernetes.io/projected/82af6c97-0661-4765-b9ee-b3928848fece-kube-api-access-lmnv4\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:29.734280 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.734242 5136 generic.go:356] "Generic (PLEG): container finished" podID="82af6c97-0661-4765-b9ee-b3928848fece" containerID="9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2" exitCode=0 Nov 02 21:17:29.734471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.734291 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" Nov 02 21:17:29.734471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.734292 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" event={"ID":"82af6c97-0661-4765-b9ee-b3928848fece","Type":"ContainerDied","Data":"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2"} Nov 02 21:17:29.734471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.734350 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh" event={"ID":"82af6c97-0661-4765-b9ee-b3928848fece","Type":"ContainerDied","Data":"b682fcbc93fb3c9d99380d10ac7ff758b08a59906fe0c24bae926d49a81427ad"} Nov 02 21:17:29.734471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.734371 5136 scope.go:122] "RemoveContainer" containerID="9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2" Nov 02 21:17:29.743236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.743216 5136 scope.go:122] "RemoveContainer" containerID="9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2" Nov 02 21:17:29.743499 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:29.743480 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2\": container with ID starting with 9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2 not found: ID does not exist" containerID="9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2" Nov 02 21:17:29.743552 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.743510 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2"} err="failed to get container status \"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2\": rpc error: code = NotFound desc = could not find container \"9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2\": container with ID starting with 9e5aaae403e06ffe5e79915ad5fe1d3d580158fdf07e13994a186e5b24b663d2 not found: ID does not exist" Nov 02 21:17:29.892796 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.892750 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 21:17:29.894290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:29.894266 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-76c687-5b845dbcb7-9qkmh"] Nov 02 21:17:30.545984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:30.545952 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82af6c97-0661-4765-b9ee-b3928848fece" path="/var/lib/kubelet/pods/82af6c97-0661-4765-b9ee-b3928848fece/volumes" Nov 02 21:17:32.259570 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:32.259539 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 21:17:34.386901 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.386867 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 21:17:34.387321 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.387089 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" podUID="173712fc-991a-44ee-8a1a-23bed389a699" containerName="maas-api" containerID="cri-o://75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1" gracePeriod=30 Nov 02 21:17:34.642093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.642042 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 21:17:34.717376 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717351 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") pod \"173712fc-991a-44ee-8a1a-23bed389a699\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " Nov 02 21:17:34.717561 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717388 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\" (UniqueName: \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\") pod \"173712fc-991a-44ee-8a1a-23bed389a699\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " Nov 02 21:17:34.717561 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717408 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\") pod \"173712fc-991a-44ee-8a1a-23bed389a699\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " Nov 02 21:17:34.717561 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717543 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\") pod \"173712fc-991a-44ee-8a1a-23bed389a699\" (UID: \"173712fc-991a-44ee-8a1a-23bed389a699\") " Nov 02 21:17:34.717805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717782 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca" pod "173712fc-991a-44ee-8a1a-23bed389a699" (UID: "173712fc-991a-44ee-8a1a-23bed389a699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:34.717926 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.717908 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173712fc-991a-44ee-8a1a-23bed389a699-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:34.719554 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.719528 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls" pod "173712fc-991a-44ee-8a1a-23bed389a699" (UID: "173712fc-991a-44ee-8a1a-23bed389a699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:34.719670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.719614 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs" pod "173712fc-991a-44ee-8a1a-23bed389a699" (UID: "173712fc-991a-44ee-8a1a-23bed389a699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:34.719670 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.719618 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s" pod "173712fc-991a-44ee-8a1a-23bed389a699" (UID: "173712fc-991a-44ee-8a1a-23bed389a699"). InnerVolumeSpecName "kube-api-access-2tg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:34.770403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.770374 5136 generic.go:356] "Generic (PLEG): container finished" podID="173712fc-991a-44ee-8a1a-23bed389a699" containerID="75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1" exitCode=0 Nov 02 21:17:34.770553 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.770432 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" Nov 02 21:17:34.770553 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.770459 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" event={"ID":"173712fc-991a-44ee-8a1a-23bed389a699","Type":"ContainerDied","Data":"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1"} Nov 02 21:17:34.770553 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.770526 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb" event={"ID":"173712fc-991a-44ee-8a1a-23bed389a699","Type":"ContainerDied","Data":"c456b17e4d6aa3f73ca533b3a0f6c15805c569d037ea5f0521c551e97163ac4b"} Nov 02 21:17:34.770684 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.770560 5136 scope.go:122] "RemoveContainer" containerID="75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1" Nov 02 21:17:34.780339 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.780322 5136 scope.go:122] "RemoveContainer" containerID="75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1" Nov 02 21:17:34.780622 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:34.780602 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1\": container with ID starting with 75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1 not found: ID does not exist" containerID="75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1" Nov 02 21:17:34.780690 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.780630 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1"} err="failed to get container status \"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1\": rpc error: code = NotFound desc = could not find container \"75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1\": container with ID starting with 75be23cd07c564f943f19f35ec690e58abcd4e844c2caa5daa5c2249aa0366a1 not found: ID does not exist" Nov 02 21:17:34.818619 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.818598 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:34.818619 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.818619 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173712fc-991a-44ee-8a1a-23bed389a699-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:34.818773 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.818630 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg4s\" (UniqueName: \"kubernetes.io/projected/173712fc-991a-44ee-8a1a-23bed389a699-kube-api-access-2tg4s\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:34.924532 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.924452 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 21:17:34.928350 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:34.928322 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6b12ae-759b77d6fb-tgkcb"] Nov 02 21:17:36.546648 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:36.546619 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173712fc-991a-44ee-8a1a-23bed389a699" path="/var/lib/kubelet/pods/173712fc-991a-44ee-8a1a-23bed389a699/volumes" Nov 02 21:17:37.260759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:37.260708 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" containerID="cri-o://84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88" gracePeriod=25 Nov 02 21:17:38.845498 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:38.845462 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:48.844653 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:48.844617 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:51.003200 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:51.003165 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:17:53.254363 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:53.254327 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 21:17:53.254791 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:53.254534 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" containerID="cri-o://171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716" gracePeriod=30 Nov 02 21:17:57.510423 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.510395 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 21:17:57.643642 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.643559 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\" (UniqueName: \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\") pod \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " Nov 02 21:17:57.643842 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.643768 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\") pod \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\" (UID: \"cbc913b4-df32-4c31-bae1-b06377d1bb9b\") " Nov 02 21:17:57.644209 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.644174 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume" pod "cbc913b4-df32-4c31-bae1-b06377d1bb9b" (UID: "cbc913b4-df32-4c31-bae1-b06377d1bb9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:57.646183 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.646153 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5" pod "cbc913b4-df32-4c31-bae1-b06377d1bb9b" (UID: "cbc913b4-df32-4c31-bae1-b06377d1bb9b"). InnerVolumeSpecName "kube-api-access-kntj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:57.745371 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.745340 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc913b4-df32-4c31-bae1-b06377d1bb9b-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.745371 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.745370 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kntj5\" (UniqueName: \"kubernetes.io/projected/cbc913b4-df32-4c31-bae1-b06377d1bb9b-kube-api-access-kntj5\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.883825 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.883792 5136 generic.go:356] "Generic (PLEG): container finished" podID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerID="84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88" exitCode=0 Nov 02 21:17:57.884003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.883829 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" event={"ID":"cbc913b4-df32-4c31-bae1-b06377d1bb9b","Type":"ContainerDied","Data":"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88"} Nov 02 21:17:57.884003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.883861 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" Nov 02 21:17:57.884003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.883876 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc" event={"ID":"cbc913b4-df32-4c31-bae1-b06377d1bb9b","Type":"ContainerDied","Data":"c8b4301dd65c6da7982a52714fc4e97f726e0e7d649b4a7844947687b87202b6"} Nov 02 21:17:57.884003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.883898 5136 scope.go:122] "RemoveContainer" containerID="84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88" Nov 02 21:17:57.892720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.892699 5136 scope.go:122] "RemoveContainer" containerID="84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88" Nov 02 21:17:57.893013 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:57.892994 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88\": container with ID starting with 84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88 not found: ID does not exist" containerID="84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88" Nov 02 21:17:57.893069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:57.893021 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88"} err="failed to get container status \"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88\": rpc error: code = NotFound desc = could not find container \"84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88\": container with ID starting with 84df434caf241d3ec1af111158e1aa4024f19678e9fac02f2ba64e2703bc4d88 not found: ID does not exist" Nov 02 21:17:58.038385 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.038356 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 21:17:58.085656 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.085629 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6b12ae-7b5c4f64b5-gfjbc"] Nov 02 21:17:58.507139 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.507118 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 21:17:58.545847 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.545792 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" path="/var/lib/kubelet/pods/cbc913b4-df32-4c31-bae1-b06377d1bb9b/volumes" Nov 02 21:17:58.655193 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655167 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655212 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655269 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655310 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655334 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655384 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655423 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655451 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655499 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\") pod \"e4b06940-d7c2-4a9b-a302-9421de92d310\" (UID: \"e4b06940-d7c2-4a9b-a302-9421de92d310\") " Nov 02 21:17:58.655816 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655575 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:58.655816 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655684 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:58.655816 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655716 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:58.655816 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655801 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:58.655985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655827 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:58.655985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655944 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.655985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655964 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.655985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655979 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.656201 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.655994 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b06940-d7c2-4a9b-a302-9421de92d310-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.656201 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.656006 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.657774 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.657720 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:58.657774 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.657760 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:17:58.657920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.657802 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "kube-api-access-jhtph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:58.657920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.657822 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token" pod "e4b06940-d7c2-4a9b-a302-9421de92d310" (UID: "e4b06940-d7c2-4a9b-a302-9421de92d310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:58.670098 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.670071 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 21:17:58.756513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.756475 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.756513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.756503 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.756513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.756513 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b06940-d7c2-4a9b-a302-9421de92d310-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.756513 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.756522 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtph\" (UniqueName: \"kubernetes.io/projected/e4b06940-d7c2-4a9b-a302-9421de92d310-kube-api-access-jhtph\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:58.890061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.889977 5136 generic.go:356] "Generic (PLEG): container finished" podID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerID="171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716" exitCode=0 Nov 02 21:17:58.890061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.890032 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" Nov 02 21:17:58.890061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.890056 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerDied","Data":"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716"} Nov 02 21:17:58.890337 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.890106 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r" event={"ID":"e4b06940-d7c2-4a9b-a302-9421de92d310","Type":"ContainerDied","Data":"6760b56330a25dd6ef94fa400cf7e18104818b6bce8d7aafa6258097c7cd45bd"} Nov 02 21:17:58.890337 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.890136 5136 scope.go:122] "RemoveContainer" containerID="171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716" Nov 02 21:17:58.900212 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.900193 5136 scope.go:122] "RemoveContainer" containerID="bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5" Nov 02 21:17:58.908160 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.908139 5136 scope.go:122] "RemoveContainer" containerID="171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716" Nov 02 21:17:58.908425 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:58.908406 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716\": container with ID starting with 171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716 not found: ID does not exist" containerID="171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716" Nov 02 21:17:58.908499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.908439 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716"} err="failed to get container status \"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716\": rpc error: code = NotFound desc = could not find container \"171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716\": container with ID starting with 171871c3d7a8338e15ea7f034838cae9af8782dc3566c4425051d93f94789716 not found: ID does not exist" Nov 02 21:17:58.908499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.908466 5136 scope.go:122] "RemoveContainer" containerID="bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5" Nov 02 21:17:58.908767 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:17:58.908718 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5\": container with ID starting with bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5 not found: ID does not exist" containerID="bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5" Nov 02 21:17:58.908810 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:58.908774 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5"} err="failed to get container status \"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5\": rpc error: code = NotFound desc = could not find container \"bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5\": container with ID starting with bc3d220eb0427af14705d07f4225f3d6bf87e75b75221781cef055614f5e11c5 not found: ID does not exist" Nov 02 21:17:59.045324 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:59.045289 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 21:17:59.048542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:17:59.048514 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6b12ae-openshift-default-6585b85975-5vx9r"] Nov 02 21:18:00.545884 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:00.545847 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" path="/var/lib/kubelet/pods/e4b06940-d7c2-4a9b-a302-9421de92d310/volumes" Nov 02 21:18:13.689860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.689816 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" containerID="cri-o://0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b" gracePeriod=45 Nov 02 21:18:13.706765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.706722 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Nov 02 21:18:13.933326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.933291 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 21:18:13.962538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.962502 5136 generic.go:356] "Generic (PLEG): container finished" podID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerID="0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b" exitCode=0 Nov 02 21:18:13.962688 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.962536 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" event={"ID":"0354fd75-2ef1-4e92-9751-dd1f945b4472","Type":"ContainerDied","Data":"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b"} Nov 02 21:18:13.962688 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.962559 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" Nov 02 21:18:13.962688 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.962596 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl" event={"ID":"0354fd75-2ef1-4e92-9751-dd1f945b4472","Type":"ContainerDied","Data":"d79a9bebaa1a1d1947ad93603cd2ea3abd2ceddf13c08b9a142261d52b3d934b"} Nov 02 21:18:13.962688 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.962619 5136 scope.go:122] "RemoveContainer" containerID="0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b" Nov 02 21:18:13.971320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.971303 5136 scope.go:122] "RemoveContainer" containerID="0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b" Nov 02 21:18:13.971570 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:13.971551 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b\": container with ID starting with 0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b not found: ID does not exist" containerID="0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b" Nov 02 21:18:13.971634 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.971582 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b"} err="failed to get container status \"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b\": rpc error: code = NotFound desc = could not find container \"0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b\": container with ID starting with 0af4c0ca075eabded8fe80ad2a53a2c1588888fa36e8761048810df7d4e6279b not found: ID does not exist" Nov 02 21:18:13.999147 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999128 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\" (UniqueName: \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999206 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999240 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\" (UniqueName: \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999347 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999266 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999347 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999299 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999468 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999355 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\") pod \"0354fd75-2ef1-4e92-9751-dd1f945b4472\" (UID: \"0354fd75-2ef1-4e92-9751-dd1f945b4472\") " Nov 02 21:18:13.999659 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999593 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:13.999761 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999651 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:13.999761 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:13.999715 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.001352 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.001332 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.001445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.001354 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:18:14.001445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.001436 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb" pod "0354fd75-2ef1-4e92-9751-dd1f945b4472" (UID: "0354fd75-2ef1-4e92-9751-dd1f945b4472"). InnerVolumeSpecName "kube-api-access-pm2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:14.100385 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100355 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.100385 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100380 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2gb\" (UniqueName: \"kubernetes.io/projected/0354fd75-2ef1-4e92-9751-dd1f945b4472-kube-api-access-pm2gb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.100385 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100390 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.100697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100398 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.100697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100407 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0354fd75-2ef1-4e92-9751-dd1f945b4472-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.100697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.100415 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0354fd75-2ef1-4e92-9751-dd1f945b4472-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.376215 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.376181 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 21:18:14.379958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.379933 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/distinct-w0-14d090-kserve-d6bb57c57-5x7tl"] Nov 02 21:18:14.550854 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:14.550774 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" path="/var/lib/kubelet/pods/0354fd75-2ef1-4e92-9751-dd1f945b4472/volumes" Nov 02 21:18:21.013854 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:21.013647 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:18:22.006073 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.006039 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 21:18:22.006301 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.006258 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" podUID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" containerName="maas-api" containerID="cri-o://3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751" gracePeriod=30 Nov 02 21:18:22.255228 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.255205 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 21:18:22.386344 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.386268 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\") pod \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " Nov 02 21:18:22.386344 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.386337 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\" (UniqueName: \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\") pod \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " Nov 02 21:18:22.386547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.386371 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") pod \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " Nov 02 21:18:22.386547 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.386426 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\") pod \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\" (UID: \"b80e468e-ea96-4557-9c0b-9ae84f6895e1\") " Nov 02 21:18:22.386870 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.386841 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca" pod "b80e468e-ea96-4557-9c0b-9ae84f6895e1" (UID: "b80e468e-ea96-4557-9c0b-9ae84f6895e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:18:22.388580 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.388551 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls" pod "b80e468e-ea96-4557-9c0b-9ae84f6895e1" (UID: "b80e468e-ea96-4557-9c0b-9ae84f6895e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:18:22.388580 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.388572 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk" pod "b80e468e-ea96-4557-9c0b-9ae84f6895e1" (UID: "b80e468e-ea96-4557-9c0b-9ae84f6895e1"). InnerVolumeSpecName "kube-api-access-55ngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:22.388707 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.388646 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs" pod "b80e468e-ea96-4557-9c0b-9ae84f6895e1" (UID: "b80e468e-ea96-4557-9c0b-9ae84f6895e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:18:22.487685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.487651 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80e468e-ea96-4557-9c0b-9ae84f6895e1-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:22.487685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.487685 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:22.487900 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.487697 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ngk\" (UniqueName: \"kubernetes.io/projected/b80e468e-ea96-4557-9c0b-9ae84f6895e1-kube-api-access-55ngk\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:22.487900 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:22.487707 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80e468e-ea96-4557-9c0b-9ae84f6895e1-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:23.010689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.010654 5136 generic.go:356] "Generic (PLEG): container finished" podID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" containerID="3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751" exitCode=0 Nov 02 21:18:23.010890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.010701 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" Nov 02 21:18:23.010890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.010745 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" event={"ID":"b80e468e-ea96-4557-9c0b-9ae84f6895e1","Type":"ContainerDied","Data":"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751"} Nov 02 21:18:23.010890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.010800 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj" event={"ID":"b80e468e-ea96-4557-9c0b-9ae84f6895e1","Type":"ContainerDied","Data":"3c58762c685b6e211ae08437bac85d7ef00d9b96e1ff10b8a10651ab1b42d5de"} Nov 02 21:18:23.010890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.010821 5136 scope.go:122] "RemoveContainer" containerID="3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751" Nov 02 21:18:23.019597 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.019578 5136 scope.go:122] "RemoveContainer" containerID="3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751" Nov 02 21:18:23.019972 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:23.019953 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751\": container with ID starting with 3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751 not found: ID does not exist" containerID="3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751" Nov 02 21:18:23.020062 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.019983 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751"} err="failed to get container status \"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751\": rpc error: code = NotFound desc = could not find container \"3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751\": container with ID starting with 3c13754c12fa87ec19dd75e61aacdf836c26951cf9b26ab4207332cae4b5b751 not found: ID does not exist" Nov 02 21:18:23.166656 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.166622 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 21:18:23.213938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:23.213908 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-14d090-577ddd4445-7plnj"] Nov 02 21:18:24.545409 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:24.545375 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" path="/var/lib/kubelet/pods/b80e468e-ea96-4557-9c0b-9ae84f6895e1/volumes" Nov 02 21:18:26.149527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:26.149496 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 21:18:26.149995 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:26.149762 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" containerID="cri-o://f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf" gracePeriod=30 Nov 02 21:18:28.023422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:28.023398 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7cwmg_ba3a6e88-66b2-4b55-a673-6260a7cbbe1a/manager/0.log" Nov 02 21:18:28.228667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:28.228638 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pppkd_0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff/kuadrant-console-plugin/0.log" Nov 02 21:18:28.970119 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:28.970078 5136 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:18:28.970302 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:28.970152 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:18:30.735811 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:30.735781 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656cc65bfd-xlcwc_78556d8e-59a8-4977-8d09-8586ee3cb6c7/manager/0.log" Nov 02 21:18:31.279622 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.279597 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mg6sc_95efd56e-e640-4249-b8f4-5e55d11f91c2/manager/0.log" Nov 02 21:18:31.396672 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.396649 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 21:18:31.570151 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570115 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570151 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570157 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570405 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570177 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570405 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570205 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570405 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570342 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570416 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570444 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570519 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:31.570767 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570592 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570767 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570653 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\") pod \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\" (UID: \"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5\") " Nov 02 21:18:31.570767 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570690 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:31.570938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570826 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:31.570999 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.570951 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:31.571172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.571150 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.571266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.571185 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.571266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.571201 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.571266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.571214 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.571266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.571151 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:18:31.572893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.572867 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:31.572893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.572878 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:31.573102 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.572981 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "kube-api-access-5lj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:31.573249 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.573226 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo" pod "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" (UID: "ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:18:31.672406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.672371 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.672406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.672400 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.672406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.672414 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.672639 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.672423 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lj6m\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-kube-api-access-5lj6m\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:31.672639 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:31.672432 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:32.056518 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.056480 5136 generic.go:356] "Generic (PLEG): container finished" podID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerID="f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf" exitCode=0 Nov 02 21:18:32.057025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.056586 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" event={"ID":"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5","Type":"ContainerDied","Data":"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf"} Nov 02 21:18:32.057025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.056590 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" Nov 02 21:18:32.057025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.056656 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv" event={"ID":"ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5","Type":"ContainerDied","Data":"b2aa8cfcbf5fc6b3957d9dffc79e5d6739567c05f6530477425e98c1de441361"} Nov 02 21:18:32.057025 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.056689 5136 scope.go:122] "RemoveContainer" containerID="f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf" Nov 02 21:18:32.066411 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.066394 5136 scope.go:122] "RemoveContainer" containerID="f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf" Nov 02 21:18:32.066657 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:32.066639 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf\": container with ID starting with f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf not found: ID does not exist" containerID="f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf" Nov 02 21:18:32.066708 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.066664 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf"} err="failed to get container status \"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf\": rpc error: code = NotFound desc = could not find container \"f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf\": container with ID starting with f21a3bb04f13912f9aa8d5a36a62e8eeb92a6c192c17a16d7302b33e1951b4bf not found: ID does not exist" Nov 02 21:18:32.195187 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.195156 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-snlvz_f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78/manager/1.log" Nov 02 21:18:32.213099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.213069 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 21:18:32.260264 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.260229 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-zs4vv"] Nov 02 21:18:32.433111 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.433039 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w9xl7_73e480fc-41e8-405f-a466-2e08f9041414/server/0.log" Nov 02 21:18:32.544997 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.544966 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" path="/var/lib/kubelet/pods/ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5/volumes" Nov 02 21:18:32.647675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.647646 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2slch_186a3e83-bf4f-4cca-972d-2761964a7c63/manager/0.log" Nov 02 21:18:32.892435 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:32.892404 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-jb6hq_78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b/proxy/0.log" Nov 02 21:18:33.501149 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:33.501114 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-snlvz_f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78/manager/0.log" Nov 02 21:18:37.965256 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965225 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965621 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" containerName="main" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965632 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" containerName="main" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965645 5136 cpu_manager.go:414] "containerMap: removing container" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965651 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965660 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1dab15d7-86a1-45a7-804c-348e82f90833" containerName="maas-api" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965666 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dab15d7-86a1-45a7-804c-348e82f90833" containerName="maas-api" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965675 5136 cpu_manager.go:414] "containerMap: removing container" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965680 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.965685 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965687 5136 cpu_manager.go:414] "containerMap: removing container" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965693 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965700 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965705 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965714 5136 cpu_manager.go:414] "containerMap: removing container" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965718 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965725 5136 cpu_manager.go:414] "containerMap: removing container" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965742 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965749 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965753 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965764 5136 cpu_manager.go:414] "containerMap: removing container" podUID="9be482bd-db18-4449-b943-8c2d93236048" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965768 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be482bd-db18-4449-b943-8c2d93236048" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965776 5136 cpu_manager.go:414] "containerMap: removing container" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965780 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965789 5136 cpu_manager.go:414] "containerMap: removing container" podUID="173712fc-991a-44ee-8a1a-23bed389a699" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965793 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173712fc-991a-44ee-8a1a-23bed389a699" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965800 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965804 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965810 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965819 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965823 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965828 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965834 5136 cpu_manager.go:414] "containerMap: removing container" podUID="34e506f6-598b-480f-876e-e7cbb08ab632" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965839 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e506f6-598b-480f-876e-e7cbb08ab632" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965846 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965850 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965857 5136 cpu_manager.go:414] "containerMap: removing container" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965862 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965873 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965877 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965882 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965886 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965894 5136 cpu_manager.go:414] "containerMap: removing container" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965898 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965908 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965912 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965918 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965922 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965930 5136 cpu_manager.go:414] "containerMap: removing container" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965934 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.965997 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0354fd75-2ef1-4e92-9751-dd1f945b4472" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966003 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7bdf44-35b1-4525-91d9-3e35fb6ec5b5" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966009 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6f95a6-eee7-4b3e-8aff-fb99a6214377" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966016 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80e468e-ea96-4557-9c0b-9ae84f6895e1" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966022 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966028 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d04e1e6-40a4-4ee4-80d0-c6ff72f91f61" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966033 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aae6438-56d6-4650-ab1b-97e08f1cded8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966040 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac130cd9-f3ab-4b01-9e38-7d7647f62468" containerName="main" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966047 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="82af6c97-0661-4765-b9ee-b3928848fece" containerName="payload-processing" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966052 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dab15d7-86a1-45a7-804c-348e82f90833" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966059 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966066 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="765c1df4-1f03-4847-b5a6-f4a075c40ee8" containerName="istio-proxy" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966071 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1297967-4c0f-4520-b2f9-fdbcfe3c5d5f" containerName="maas-api" Nov 02 21:18:37.966051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966077 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4f3c8-0d3d-4fd9-a44e-9e6c90461c01" containerName="istio-proxy" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966082 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8adf00-4cf7-4663-ae94-8f70534eb244" containerName="istio-proxy" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966088 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966093 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be482bd-db18-4449-b943-8c2d93236048" containerName="main" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966098 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f903d4-b68e-4291-869f-2092f360a459" containerName="main" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966103 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="173712fc-991a-44ee-8a1a-23bed389a699" containerName="maas-api" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966108 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d758097-e052-4a9e-b5ee-4ee30c33b38f" containerName="maas-api" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966113 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b06940-d7c2-4a9b-a302-9421de92d310" containerName="istio-proxy" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966119 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e506f6-598b-480f-876e-e7cbb08ab632" containerName="maas-api" Nov 02 21:18:37.967920 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.966123 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc913b4-df32-4c31-bae1-b06377d1bb9b" containerName="payload-processing" Nov 02 21:18:37.970495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:37.970468 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.037885 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.037856 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-056961db-openshift-default-dockercfg-lb92q\"" Nov 02 21:18:38.069766 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.069721 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:18:38.130145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130115 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130353 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130155 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130353 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130261 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130353 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130302 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130353 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130327 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldkp\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130528 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130435 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130528 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130498 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130610 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130535 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.130610 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.130554 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231231 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldkp\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231269 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231397 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231442 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231476 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231531 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231819 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231572 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231933 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.231959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.231954 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.232119 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.232043 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.232319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.232294 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.232422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.232335 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.232572 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.232550 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.232681 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.232656 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.234216 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.234194 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.234367 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.234347 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.285472 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.285445 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.285472 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.285458 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldkp\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\") pod \"e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.583687 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.583606 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:38.995310 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.995288 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:18:38.997536 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:18:38.997510 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64548c2e_941d_4572_bb68_c30110f4a741.slice/crio-228138738fea6bc4054d38874b73043508c122fe26fcf1d58e381e5dc29fc11a WatchSource:0}: Error finding container 228138738fea6bc4054d38874b73043508c122fe26fcf1d58e381e5dc29fc11a: Status 404 returned error can't find the container with id 228138738fea6bc4054d38874b73043508c122fe26fcf1d58e381e5dc29fc11a Nov 02 21:18:38.999714 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.999674 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:18:38.999815 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.999776 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:18:38.999859 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:38.999817 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:18:39.092907 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:39.092870 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerStarted","Data":"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2"} Nov 02 21:18:39.093076 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:39.092926 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerStarted","Data":"228138738fea6bc4054d38874b73043508c122fe26fcf1d58e381e5dc29fc11a"} Nov 02 21:18:39.204410 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:39.204355 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podStartSLOduration=2.204341528 podStartE2EDuration="2.204341528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:39.20305087 +0000 UTC m=+5402.809595372" watchObservedRunningTime="2026-11-02 21:18:39.204341528 +0000 UTC m=+5402.810886026" Nov 02 21:18:39.584469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:39.584439 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:40.589284 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:40.589249 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:41.102026 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:41.101997 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:41.102961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:41.102944 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:18:44.784974 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.784945 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:18:44.785409 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.785394 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9619b5-758e-4ae0-a88d-d0ea4cd17988" containerName="istio-proxy" Nov 02 21:18:44.788558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.788541 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:44.838703 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.838662 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-056961db\"" Nov 02 21:18:44.883787 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.883709 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:44.883949 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.883824 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntk9q\" (UniqueName: \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:44.886463 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.886432 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-056961db-dockercfg-6mjj9\"" Nov 02 21:18:44.888037 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.888018 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:18:44.944910 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.944878 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:18:44.948778 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.948755 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:44.985049 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.985027 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:44.985188 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.985090 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntk9q\" (UniqueName: \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:44.985188 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.985135 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:44.985188 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.985176 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7l6\" (UniqueName: \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:44.985649 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:44.985628 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:45.001084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.001062 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:18:45.039583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.039525 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntk9q\" (UniqueName: \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\") pod \"payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:45.085674 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.085652 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:45.085843 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.085686 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7l6\" (UniqueName: \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:45.086439 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.086417 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:45.101674 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.101652 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:45.139568 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.139530 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7l6\" (UniqueName: \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\") pod \"payload-processing-e2e-ait-056961db-5596bc5c77-7st5g\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:45.261463 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.261428 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:45.463935 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.463907 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:18:45.466112 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:18:45.466087 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8306504_f916_4473_b7df_b480edc2e476.slice/crio-a83632ff9f425472e7954247bc7453a02924dcf36c0c71ae052e27daeb0b17fd WatchSource:0}: Error finding container a83632ff9f425472e7954247bc7453a02924dcf36c0c71ae052e27daeb0b17fd: Status 404 returned error can't find the container with id a83632ff9f425472e7954247bc7453a02924dcf36c0c71ae052e27daeb0b17fd Nov 02 21:18:45.633282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:45.630413 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:18:46.127905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.127874 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" event={"ID":"800997b0-e774-48d0-b16f-38cae1afa9a0","Type":"ContainerStarted","Data":"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774"} Nov 02 21:18:46.128389 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.127925 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" event={"ID":"800997b0-e774-48d0-b16f-38cae1afa9a0","Type":"ContainerStarted","Data":"2ea0301a7e86b8dc4150bd45d59912fd57cf45ec590adccccd0782d574446def"} Nov 02 21:18:46.128389 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.127952 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:46.129364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.129334 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" event={"ID":"f8306504-f916-4473-b7df-b480edc2e476","Type":"ContainerStarted","Data":"56582ec8488ea4680f42cbd31e72f8702cee204cbb1c6046e0541fc5b3f6ee70"} Nov 02 21:18:46.129497 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.129382 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" event={"ID":"f8306504-f916-4473-b7df-b480edc2e476","Type":"ContainerStarted","Data":"a83632ff9f425472e7954247bc7453a02924dcf36c0c71ae052e27daeb0b17fd"} Nov 02 21:18:46.129497 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.129402 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:18:46.235148 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.235105 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" podStartSLOduration=2.235093901 podStartE2EDuration="2.235093901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:46.234857702 +0000 UTC m=+5409.841402201" watchObservedRunningTime="2026-11-02 21:18:46.235093901 +0000 UTC m=+5409.841638398" Nov 02 21:18:46.343143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:46.343082 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" podStartSLOduration=2.343063789 podStartE2EDuration="2.343063789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:46.341315027 +0000 UTC m=+5409.947859528" watchObservedRunningTime="2026-11-02 21:18:46.343063789 +0000 UTC m=+5409.949608284" Nov 02 21:18:47.451672 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.451641 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:18:47.455428 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.455407 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.506691 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.506663 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng65\" (UniqueName: \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.506886 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.506823 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.507157 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.507130 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.507266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.507237 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.554214 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.554182 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-056961db-serving-cert\"" Nov 02 21:18:47.555895 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.555865 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:18:47.608295 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.608257 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.608295 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.608299 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.608538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.608338 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.608538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.608418 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng65\" (UniqueName: \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.608538 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:47.608474 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-056961db-serving-cert: secret "maas-api-e2e-ait-056961db-serving-cert" not found Nov 02 21:18:47.608538 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:47.608538 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls podName:b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f nodeName:}" failed. No retries permitted until 2026-11-02 21:18:48.108517092 +0000 UTC m=+5411.715061568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls") pod "maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" (UID: "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f") : secret "maas-api-e2e-ait-056961db-serving-cert" not found Nov 02 21:18:47.609176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.609155 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.610864 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.610845 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:47.662830 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:47.662806 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng65\" (UniqueName: \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:48.113482 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:48.113449 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:48.115957 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:48.115934 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") pod \"maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:48.368956 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:48.368875 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:48.732979 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:48.732953 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:18:48.734870 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:18:48.734832 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d4b6ab_2db2_4dac_b0b2_4d3fe531232f.slice/crio-ddc1d9ffe76845dc4ed8fe624f0d436a1993599c8e2823a46233af594aed4693 WatchSource:0}: Error finding container ddc1d9ffe76845dc4ed8fe624f0d436a1993599c8e2823a46233af594aed4693: Status 404 returned error can't find the container with id ddc1d9ffe76845dc4ed8fe624f0d436a1993599c8e2823a46233af594aed4693 Nov 02 21:18:48.736405 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:48.736384 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:18:49.145211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:49.145134 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" event={"ID":"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f","Type":"ContainerStarted","Data":"ddc1d9ffe76845dc4ed8fe624f0d436a1993599c8e2823a46233af594aed4693"} Nov 02 21:18:50.150681 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:50.150642 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" event={"ID":"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f","Type":"ContainerStarted","Data":"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9"} Nov 02 21:18:50.151133 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:50.150790 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:50.308018 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:50.307967 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" podStartSLOduration=2.426426516 podStartE2EDuration="3.307949332s" totalImagesPullingTime="881.522816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:50.259960881 +0000 UTC m=+5413.866505379" watchObservedRunningTime="2026-11-02 21:18:50.307949332 +0000 UTC m=+5413.914493831" Nov 02 21:18:51.003147 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:18:51.003110 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:18:56.169134 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:56.169105 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:18:57.137143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:57.137112 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:18:57.137337 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:18:57.137180 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:19:06.254958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.254924 5136 generic.go:356] "Generic (PLEG): container finished" podID="64548c2e-941d-4572-bb68-c30110f4a741" containerID="1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2" exitCode=0 Nov 02 21:19:06.255471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.254996 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerDied","Data":"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2"} Nov 02 21:19:06.255471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.255350 5136 scope.go:122] "RemoveContainer" containerID="1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2" Nov 02 21:19:06.256478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.256449 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:19:06.256578 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.256516 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:19:06.256578 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:06.256545 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:19:07.261299 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:07.261263 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerStarted","Data":"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc"} Nov 02 21:19:07.583963 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:07.583889 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:07.585138 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:07.585110 5136 patch_prober.go:29] interesting pod/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 02 21:19:07.585268 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:07.585156 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 02 21:19:08.584748 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:08.584709 5136 patch_prober.go:29] interesting pod/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 02 21:19:08.585170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:08.584784 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 02 21:19:09.584612 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:09.584576 5136 patch_prober.go:29] interesting pod/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 02 21:19:09.584825 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:09.584644 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 02 21:19:10.584620 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:10.584580 5136 patch_prober.go:29] interesting pod/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 02 21:19:10.584864 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:10.584653 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 02 21:19:11.108760 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:11.107832 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:11.584214 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:11.584177 5136 patch_prober.go:29] interesting pod/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 02 21:19:11.584403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:11.584229 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 02 21:19:12.588302 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:12.588269 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:12.589197 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:12.589174 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:21.003862 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:21.003826 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:19:29.109980 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:29.109946 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:19:29.209666 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:29.209630 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:19:34.110354 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.110312 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" podUID="800997b0-e774-48d0-b16f-38cae1afa9a0" containerName="payload-processing" containerID="cri-o://176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774" gracePeriod=25 Nov 02 21:19:34.210336 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.210268 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" podUID="f8306504-f916-4473-b7df-b480edc2e476" containerName="payload-pre-processing" containerID="cri-o://56582ec8488ea4680f42cbd31e72f8702cee204cbb1c6046e0541fc5b3f6ee70" gracePeriod=25 Nov 02 21:19:34.350565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.350539 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:19:34.395976 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.395935 5136 generic.go:356] "Generic (PLEG): container finished" podID="f8306504-f916-4473-b7df-b480edc2e476" containerID="56582ec8488ea4680f42cbd31e72f8702cee204cbb1c6046e0541fc5b3f6ee70" exitCode=0 Nov 02 21:19:34.396144 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.396009 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" event={"ID":"f8306504-f916-4473-b7df-b480edc2e476","Type":"ContainerDied","Data":"56582ec8488ea4680f42cbd31e72f8702cee204cbb1c6046e0541fc5b3f6ee70"} Nov 02 21:19:34.397465 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.397441 5136 generic.go:356] "Generic (PLEG): container finished" podID="800997b0-e774-48d0-b16f-38cae1afa9a0" containerID="176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774" exitCode=0 Nov 02 21:19:34.397582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.397485 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" event={"ID":"800997b0-e774-48d0-b16f-38cae1afa9a0","Type":"ContainerDied","Data":"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774"} Nov 02 21:19:34.397582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.397493 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" Nov 02 21:19:34.397582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.397528 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g" event={"ID":"800997b0-e774-48d0-b16f-38cae1afa9a0","Type":"ContainerDied","Data":"2ea0301a7e86b8dc4150bd45d59912fd57cf45ec590adccccd0782d574446def"} Nov 02 21:19:34.397582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.397551 5136 scope.go:122] "RemoveContainer" containerID="176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774" Nov 02 21:19:34.407923 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.407862 5136 scope.go:122] "RemoveContainer" containerID="176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774" Nov 02 21:19:34.408169 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:34.408150 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774\": container with ID starting with 176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774 not found: ID does not exist" containerID="176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774" Nov 02 21:19:34.408255 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.408178 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774"} err="failed to get container status \"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774\": rpc error: code = NotFound desc = could not find container \"176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774\": container with ID starting with 176ae93c95ca2e54d4a8cb4d278e83fe8d846d7bd4f34299644b920f0fe86774 not found: ID does not exist" Nov 02 21:19:34.450702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.450682 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:19:34.475559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.475527 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\") pod \"800997b0-e774-48d0-b16f-38cae1afa9a0\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " Nov 02 21:19:34.475709 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.475692 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\" (UniqueName: \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\") pod \"800997b0-e774-48d0-b16f-38cae1afa9a0\" (UID: \"800997b0-e774-48d0-b16f-38cae1afa9a0\") " Nov 02 21:19:34.475905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.475884 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume" pod "800997b0-e774-48d0-b16f-38cae1afa9a0" (UID: "800997b0-e774-48d0-b16f-38cae1afa9a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:34.476122 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.476097 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800997b0-e774-48d0-b16f-38cae1afa9a0-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:34.477754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.477718 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6" pod "800997b0-e774-48d0-b16f-38cae1afa9a0" (UID: "800997b0-e774-48d0-b16f-38cae1afa9a0"). InnerVolumeSpecName "kube-api-access-lc7l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:34.576919 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.576890 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\" (UniqueName: \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\") pod \"f8306504-f916-4473-b7df-b480edc2e476\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " Nov 02 21:19:34.577077 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.576941 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\") pod \"f8306504-f916-4473-b7df-b480edc2e476\" (UID: \"f8306504-f916-4473-b7df-b480edc2e476\") " Nov 02 21:19:34.577172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.577159 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7l6\" (UniqueName: \"kubernetes.io/projected/800997b0-e774-48d0-b16f-38cae1afa9a0-kube-api-access-lc7l6\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:34.577317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.577293 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume" pod "f8306504-f916-4473-b7df-b480edc2e476" (UID: "f8306504-f916-4473-b7df-b480edc2e476"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:34.579004 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.578982 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q" pod "f8306504-f916-4473-b7df-b480edc2e476" (UID: "f8306504-f916-4473-b7df-b480edc2e476"). InnerVolumeSpecName "kube-api-access-ntk9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:34.678540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.678457 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntk9q\" (UniqueName: \"kubernetes.io/projected/f8306504-f916-4473-b7df-b480edc2e476-kube-api-access-ntk9q\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:34.678540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.678489 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8306504-f916-4473-b7df-b480edc2e476-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:34.805577 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.805541 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:19:34.808673 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:34.808649 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-056961db-5596bc5c77-7st5g"] Nov 02 21:19:35.403828 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:35.403716 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" event={"ID":"f8306504-f916-4473-b7df-b480edc2e476","Type":"ContainerDied","Data":"a83632ff9f425472e7954247bc7453a02924dcf36c0c71ae052e27daeb0b17fd"} Nov 02 21:19:35.403828 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:35.403797 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd" Nov 02 21:19:35.404321 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:35.403842 5136 scope.go:122] "RemoveContainer" containerID="56582ec8488ea4680f42cbd31e72f8702cee204cbb1c6046e0541fc5b3f6ee70" Nov 02 21:19:35.557356 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:35.557322 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:19:35.604788 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:35.604756 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-056961db-79f5bc794c-wr2jd"] Nov 02 21:19:36.544853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:36.544820 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800997b0-e774-48d0-b16f-38cae1afa9a0" path="/var/lib/kubelet/pods/800997b0-e774-48d0-b16f-38cae1afa9a0/volumes" Nov 02 21:19:36.545249 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:36.545211 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8306504-f916-4473-b7df-b480edc2e476" path="/var/lib/kubelet/pods/f8306504-f916-4473-b7df-b480edc2e476/volumes" Nov 02 21:19:36.897192 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:36.897114 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:19:36.897368 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:36.897323 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" podUID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" containerName="maas-api" containerID="cri-o://4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9" gracePeriod=30 Nov 02 21:19:37.148641 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.148585 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:19:37.305934 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.305904 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\" (UniqueName: \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\") pod \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " Nov 02 21:19:37.306099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.305991 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\") pod \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " Nov 02 21:19:37.306099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.306015 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") pod \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " Nov 02 21:19:37.306099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.306038 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\") pod \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\" (UID: \"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f\") " Nov 02 21:19:37.306511 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.306468 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca" pod "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" (UID: "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:37.308158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.308129 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65" pod "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" (UID: "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f"). InnerVolumeSpecName "kube-api-access-9ng65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:37.308158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.308143 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs" pod "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" (UID: "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:37.308313 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.308174 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls" pod "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" (UID: "b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:37.407838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.407759 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ng65\" (UniqueName: \"kubernetes.io/projected/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-kube-api-access-9ng65\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:37.407838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.407787 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:37.407838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.407798 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:37.407838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.407807 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:37.416476 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.416448 5136 generic.go:356] "Generic (PLEG): container finished" podID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" containerID="4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9" exitCode=0 Nov 02 21:19:37.416626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.416515 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" Nov 02 21:19:37.416626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.416535 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" event={"ID":"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f","Type":"ContainerDied","Data":"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9"} Nov 02 21:19:37.416626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.416589 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz" event={"ID":"b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f","Type":"ContainerDied","Data":"ddc1d9ffe76845dc4ed8fe624f0d436a1993599c8e2823a46233af594aed4693"} Nov 02 21:19:37.416626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.416612 5136 scope.go:122] "RemoveContainer" containerID="4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9" Nov 02 21:19:37.425569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.425554 5136 scope.go:122] "RemoveContainer" containerID="4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9" Nov 02 21:19:37.425875 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:37.425854 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9\": container with ID starting with 4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9 not found: ID does not exist" containerID="4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9" Nov 02 21:19:37.425946 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.425890 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9"} err="failed to get container status \"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9\": rpc error: code = NotFound desc = could not find container \"4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9\": container with ID starting with 4778fcf01572f19c07e32a61d49c462e0203d3f992301e740d2e2f45ab6094b9 not found: ID does not exist" Nov 02 21:19:37.570471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.570443 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:19:37.572244 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:37.572220 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-056961db-58f6f4b4d-ftcdz"] Nov 02 21:19:38.545720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:38.545687 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" path="/var/lib/kubelet/pods/b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f/volumes" Nov 02 21:19:43.829700 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:43.829668 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:19:43.830166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:43.829880 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" containerID="cri-o://031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc" gracePeriod=30 Nov 02 21:19:48.097080 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.097045 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 21:19:48.097515 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.097239 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-94dcf6f94-snlvz" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" containerID="cri-o://61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d" gracePeriod=30 Nov 02 21:19:48.445322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.445301 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 21:19:48.470504 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.470470 5136 generic.go:356] "Generic (PLEG): container finished" podID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerID="61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d" exitCode=0 Nov 02 21:19:48.470615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.470516 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-snlvz" Nov 02 21:19:48.470615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.470554 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerDied","Data":"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d"} Nov 02 21:19:48.470615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.470614 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-snlvz" event={"ID":"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78","Type":"ContainerDied","Data":"0cd09905b4a19dfab092f7cd52cc80e1a8e35f15c5a5f88f7d1ccfcda77a9b90"} Nov 02 21:19:48.470780 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.470637 5136 scope.go:122] "RemoveContainer" containerID="61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d" Nov 02 21:19:48.479579 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.479563 5136 scope.go:122] "RemoveContainer" containerID="70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af" Nov 02 21:19:48.486889 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.486872 5136 scope.go:122] "RemoveContainer" containerID="61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d" Nov 02 21:19:48.487175 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:48.487157 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d\": container with ID starting with 61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d not found: ID does not exist" containerID="61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d" Nov 02 21:19:48.487227 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.487185 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d"} err="failed to get container status \"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d\": rpc error: code = NotFound desc = could not find container \"61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d\": container with ID starting with 61f21b3474ddb3cd23204c1f36c9c986818a451c86735c60a47634724e54222d not found: ID does not exist" Nov 02 21:19:48.487227 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.487202 5136 scope.go:122] "RemoveContainer" containerID="70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af" Nov 02 21:19:48.487454 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:48.487436 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af\": container with ID starting with 70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af not found: ID does not exist" containerID="70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af" Nov 02 21:19:48.487504 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.487461 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af"} err="failed to get container status \"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af\": rpc error: code = NotFound desc = could not find container \"70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af\": container with ID starting with 70a2be6702815368ea3f659ded2bb804a14ad5316ba9b507c8e9b8266fd391af not found: ID does not exist" Nov 02 21:19:48.509967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.509946 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\") pod \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " Nov 02 21:19:48.510053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.510022 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\" (UniqueName: \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\") pod \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " Nov 02 21:19:48.510141 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.510124 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\") pod \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\" (UID: \"f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78\") " Nov 02 21:19:48.511962 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.511934 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs" pod "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" (UID: "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:48.512065 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.512009 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb" pod "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" (UID: "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78"). InnerVolumeSpecName "kube-api-access-gpbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:48.512126 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.512063 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert" pod "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" (UID: "f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:48.611592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.611550 5136 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-webhook-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:48.611592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.611583 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbmb\" (UniqueName: \"kubernetes.io/projected/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-kube-api-access-gpbmb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:48.611592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.611598 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:48.878027 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.877997 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 21:19:48.882838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:48.882813 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-snlvz"] Nov 02 21:19:49.063050 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.063027 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:49.117971 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.117910 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.117975 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118007 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118056 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118104 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118261 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118299 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118297 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:19:49.118364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118314 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118410 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118418 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118480 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\") pod \"64548c2e-941d-4572-bb68-c30110f4a741\" (UID: \"64548c2e-941d-4572-bb68-c30110f4a741\") " Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118515 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118668 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118704 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118722 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64548c2e-941d-4572-bb68-c30110f4a741-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118761 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.118813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.118774 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.120369 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.120344 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:19:49.120516 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.120502 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "kube-api-access-xldkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:49.120659 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.120639 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:19:49.120719 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.120657 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token" pod "64548c2e-941d-4572-bb68-c30110f4a741" (UID: "64548c2e-941d-4572-bb68-c30110f4a741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:49.219771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.219596 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldkp\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-kube-api-access-xldkp\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.219771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.219620 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.219771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.219632 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64548c2e-941d-4572-bb68-c30110f4a741-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.219771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.219643 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64548c2e-941d-4572-bb68-c30110f4a741-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.219771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.219652 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64548c2e-941d-4572-bb68-c30110f4a741-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:49.478881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.478845 5136 generic.go:356] "Generic (PLEG): container finished" podID="64548c2e-941d-4572-bb68-c30110f4a741" containerID="031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc" exitCode=0 Nov 02 21:19:49.479085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.478901 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" Nov 02 21:19:49.479085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.478909 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerDied","Data":"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc"} Nov 02 21:19:49.479085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.478949 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s" event={"ID":"64548c2e-941d-4572-bb68-c30110f4a741","Type":"ContainerDied","Data":"228138738fea6bc4054d38874b73043508c122fe26fcf1d58e381e5dc29fc11a"} Nov 02 21:19:49.479085 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.478972 5136 scope.go:122] "RemoveContainer" containerID="031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc" Nov 02 21:19:49.487677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.487662 5136 scope.go:122] "RemoveContainer" containerID="1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2" Nov 02 21:19:49.495618 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.495600 5136 scope.go:122] "RemoveContainer" containerID="031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc" Nov 02 21:19:49.495889 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:49.495871 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc\": container with ID starting with 031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc not found: ID does not exist" containerID="031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc" Nov 02 21:19:49.495953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.495896 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc"} err="failed to get container status \"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc\": rpc error: code = NotFound desc = could not find container \"031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc\": container with ID starting with 031d800b4029f581630cc72d24546cb6107b9fbdd8c62dce4da7a114bb2458cc not found: ID does not exist" Nov 02 21:19:49.495953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.495915 5136 scope.go:122] "RemoveContainer" containerID="1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2" Nov 02 21:19:49.496182 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:49.496167 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2\": container with ID starting with 1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2 not found: ID does not exist" containerID="1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2" Nov 02 21:19:49.496237 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.496187 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2"} err="failed to get container status \"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2\": rpc error: code = NotFound desc = could not find container \"1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2\": container with ID starting with 1270922ebee7d823a3e38e698276b55df89062bee6c1ea02a102d5985fc649d2 not found: ID does not exist" Nov 02 21:19:49.633631 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.633595 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:19:49.637088 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.637058 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-056961db-openshift-default-5b88d999f9-ggz4s"] Nov 02 21:19:49.702140 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702113 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:19:49.702488 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702476 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702490 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702501 5136 cpu_manager.go:414] "containerMap: removing container" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702507 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702517 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" containerName="maas-api" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702522 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" containerName="maas-api" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702534 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f8306504-f916-4473-b7df-b480edc2e476" containerName="payload-pre-processing" Nov 02 21:19:49.702537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702539 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8306504-f916-4473-b7df-b480edc2e476" containerName="payload-pre-processing" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702554 5136 cpu_manager.go:414] "containerMap: removing container" podUID="800997b0-e774-48d0-b16f-38cae1afa9a0" containerName="payload-processing" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702560 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800997b0-e774-48d0-b16f-38cae1afa9a0" containerName="payload-processing" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702621 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702629 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8306504-f916-4473-b7df-b480edc2e476" containerName="payload-pre-processing" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702636 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702643 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="800997b0-e774-48d0-b16f-38cae1afa9a0" containerName="payload-processing" Nov 02 21:19:49.702792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.702653 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d4b6ab-2db2-4dac-b0b2-4d3fe531232f" containerName="maas-api" Nov 02 21:19:49.706034 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.706013 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.758318 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.758254 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 21:19:49.758715 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.758583 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lfcl7\"" Nov 02 21:19:49.758715 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.758678 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 21:19:49.758912 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.758796 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 21:19:49.760045 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.760024 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:19:49.824490 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.824463 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szkj\" (UniqueName: \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.824627 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.824506 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.824686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.824665 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.925792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.925764 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.925982 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.925814 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szkj\" (UniqueName: \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.925982 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.925847 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.928369 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.928345 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.928487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.928403 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:49.979626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:49.979593 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szkj\" (UniqueName: \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\") pod \"maas-controller-94dcf6f94-czmdj\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:50.018754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:50.018686 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:50.378926 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:50.378900 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:19:50.380915 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:19:50.380883 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd72c63d_3789_4b04_938f_9ab4522f8633.slice/crio-cc627505ea341ba60b21e4f12592abb6031065a89d97be0261278634e5a84811 WatchSource:0}: Error finding container cc627505ea341ba60b21e4f12592abb6031065a89d97be0261278634e5a84811: Status 404 returned error can't find the container with id cc627505ea341ba60b21e4f12592abb6031065a89d97be0261278634e5a84811 Nov 02 21:19:50.485664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:50.485632 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-czmdj" event={"ID":"fd72c63d-3789-4b04-938f-9ab4522f8633","Type":"ContainerStarted","Data":"cc627505ea341ba60b21e4f12592abb6031065a89d97be0261278634e5a84811"} Nov 02 21:19:50.544714 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:50.544691 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64548c2e-941d-4572-bb68-c30110f4a741" path="/var/lib/kubelet/pods/64548c2e-941d-4572-bb68-c30110f4a741/volumes" Nov 02 21:19:50.545113 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:50.545100 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" path="/var/lib/kubelet/pods/f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78/volumes" Nov 02 21:19:51.003876 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:19:51.003844 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:19:51.491648 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:51.491614 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-czmdj" event={"ID":"fd72c63d-3789-4b04-938f-9ab4522f8633","Type":"ContainerStarted","Data":"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f"} Nov 02 21:19:51.492041 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:51.491700 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:19:51.637972 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:19:51.637879 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-94dcf6f94-czmdj" podStartSLOduration=1.645000069 podStartE2EDuration="2.637862921s" totalImagesPullingTime="992.862852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:51.636107131 +0000 UTC m=+5475.242651629" watchObservedRunningTime="2026-11-02 21:19:51.637862921 +0000 UTC m=+5475.244407419" Nov 02 21:20:02.500875 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:02.500841 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:20:21.002770 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:20:21.002680 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:20:23.816427 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:23.816343 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:20:23.816910 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:23.816538 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-94dcf6f94-czmdj" podUID="fd72c63d-3789-4b04-938f-9ab4522f8633" containerName="manager" containerID="cri-o://bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f" gracePeriod=30 Nov 02 21:20:24.058283 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.058259 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:20:24.130038 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.129959 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\") pod \"fd72c63d-3789-4b04-938f-9ab4522f8633\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " Nov 02 21:20:24.130038 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.130011 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\" (UniqueName: \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\") pod \"fd72c63d-3789-4b04-938f-9ab4522f8633\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " Nov 02 21:20:24.130286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.130048 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\") pod \"fd72c63d-3789-4b04-938f-9ab4522f8633\" (UID: \"fd72c63d-3789-4b04-938f-9ab4522f8633\") " Nov 02 21:20:24.132144 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.132106 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert" pod "fd72c63d-3789-4b04-938f-9ab4522f8633" (UID: "fd72c63d-3789-4b04-938f-9ab4522f8633"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:24.132253 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.132206 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj" pod "fd72c63d-3789-4b04-938f-9ab4522f8633" (UID: "fd72c63d-3789-4b04-938f-9ab4522f8633"). InnerVolumeSpecName "kube-api-access-2szkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:24.132253 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.132230 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs" pod "fd72c63d-3789-4b04-938f-9ab4522f8633" (UID: "fd72c63d-3789-4b04-938f-9ab4522f8633"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:24.230819 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.230785 5136 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-webhook-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:24.230819 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.230817 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szkj\" (UniqueName: \"kubernetes.io/projected/fd72c63d-3789-4b04-938f-9ab4522f8633-kube-api-access-2szkj\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:24.231029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.230833 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd72c63d-3789-4b04-938f-9ab4522f8633-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:24.638657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.638625 5136 generic.go:356] "Generic (PLEG): container finished" podID="fd72c63d-3789-4b04-938f-9ab4522f8633" containerID="bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f" exitCode=0 Nov 02 21:20:24.638860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.638664 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-czmdj" event={"ID":"fd72c63d-3789-4b04-938f-9ab4522f8633","Type":"ContainerDied","Data":"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f"} Nov 02 21:20:24.638860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.638672 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-czmdj" Nov 02 21:20:24.638860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.638704 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-czmdj" event={"ID":"fd72c63d-3789-4b04-938f-9ab4522f8633","Type":"ContainerDied","Data":"cc627505ea341ba60b21e4f12592abb6031065a89d97be0261278634e5a84811"} Nov 02 21:20:24.638860 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.638725 5136 scope.go:122] "RemoveContainer" containerID="bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f" Nov 02 21:20:24.647535 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.647514 5136 scope.go:122] "RemoveContainer" containerID="bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f" Nov 02 21:20:24.647817 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:20:24.647796 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f\": container with ID starting with bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f not found: ID does not exist" containerID="bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f" Nov 02 21:20:24.647899 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.647824 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f"} err="failed to get container status \"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f\": rpc error: code = NotFound desc = could not find container \"bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f\": container with ID starting with bd2034ca33392bbb319b9e0cd4cec00dc3f697653f5e58de755f7f7aa21c6c4f not found: ID does not exist" Nov 02 21:20:24.794070 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.794036 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:20:24.841508 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:24.841424 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-czmdj"] Nov 02 21:20:26.132116 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132083 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94dcf6f94-5f2xx"] Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132468 5136 cpu_manager.go:414] "containerMap: removing container" podUID="fd72c63d-3789-4b04-938f-9ab4522f8633" containerName="manager" Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132481 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd72c63d-3789-4b04-938f-9ab4522f8633" containerName="manager" Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132500 5136 cpu_manager.go:414] "containerMap: removing container" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132504 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132514 5136 cpu_manager.go:414] "containerMap: removing container" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:20:26.132529 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132519 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:20:26.132783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132583 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8be4d9a-6bc3-4c08-8ff0-c6bff5e3ca78" containerName="manager" Nov 02 21:20:26.132783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132591 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="64548c2e-941d-4572-bb68-c30110f4a741" containerName="istio-proxy" Nov 02 21:20:26.132783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.132600 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd72c63d-3789-4b04-938f-9ab4522f8633" containerName="manager" Nov 02 21:20:26.137113 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.137093 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.145998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.145976 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-webhook-cert\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.146084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.146051 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-metrics-certs\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.146126 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.146080 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dfz\" (UniqueName: \"kubernetes.io/projected/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-kube-api-access-k7dfz\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.234658 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.234626 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 21:20:26.234658 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.234648 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lfcl7\"" Nov 02 21:20:26.234658 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.234629 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 21:20:26.234988 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.234704 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 21:20:26.236028 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.235996 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-5f2xx"] Nov 02 21:20:26.246528 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.246501 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-webhook-cert\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.246655 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.246545 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-metrics-certs\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.246655 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.246576 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dfz\" (UniqueName: \"kubernetes.io/projected/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-kube-api-access-k7dfz\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.249342 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.249319 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-metrics-certs\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.249444 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.249326 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-webhook-cert\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.300565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.300526 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dfz\" (UniqueName: \"kubernetes.io/projected/c157ba0d-6496-4ec6-bfad-aafbaf9439f3-kube-api-access-k7dfz\") pod \"maas-controller-94dcf6f94-5f2xx\" (UID: \"c157ba0d-6496-4ec6-bfad-aafbaf9439f3\") " pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.450081 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.450046 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:26.545309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.545277 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd72c63d-3789-4b04-938f-9ab4522f8633" path="/var/lib/kubelet/pods/fd72c63d-3789-4b04-938f-9ab4522f8633/volumes" Nov 02 21:20:26.811678 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:26.811648 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcf6f94-5f2xx"] Nov 02 21:20:26.814490 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:20:26.814447 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc157ba0d_6496_4ec6_bfad_aafbaf9439f3.slice/crio-e6472ef76d12906764cc2227f26f563108f9421c2bb23db474e8d83c3d6cf35a WatchSource:0}: Error finding container e6472ef76d12906764cc2227f26f563108f9421c2bb23db474e8d83c3d6cf35a: Status 404 returned error can't find the container with id e6472ef76d12906764cc2227f26f563108f9421c2bb23db474e8d83c3d6cf35a Nov 02 21:20:27.655249 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:27.655219 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" event={"ID":"c157ba0d-6496-4ec6-bfad-aafbaf9439f3","Type":"ContainerStarted","Data":"e6472ef76d12906764cc2227f26f563108f9421c2bb23db474e8d83c3d6cf35a"} Nov 02 21:20:28.661010 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:28.660970 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" event={"ID":"c157ba0d-6496-4ec6-bfad-aafbaf9439f3","Type":"ContainerStarted","Data":"fbd8eca413d9fb2268a088ad6218db08a8bf3aa9987a5eb26c5f6f5f7ddcac37"} Nov 02 21:20:28.661461 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:28.661130 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:28.770479 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:28.770426 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" podStartSLOduration=1.922616134 podStartE2EDuration="2.770412779s" totalImagesPullingTime="847.796645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:28.769200785 +0000 UTC m=+5512.375745283" watchObservedRunningTime="2026-11-02 21:20:28.770412779 +0000 UTC m=+5512.376957301" Nov 02 21:20:39.668690 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:20:39.668658 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94dcf6f94-5f2xx" Nov 02 21:20:51.003447 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:20:51.003407 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:21:21.002859 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:21:21.002822 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:21:24.847641 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.847602 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh"] Nov 02 21:21:24.851066 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.851046 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.949950 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.949914 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh"] Nov 02 21:21:24.950326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950300 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa795690-3689-45c2-b455-a1b7b58e873d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950416 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950363 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqs5\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-kube-api-access-zdqs5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950481 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950457 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa795690-3689-45c2-b455-a1b7b58e873d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950574 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950683 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950656 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950763 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950708 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950763 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950753 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:24.950853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:24.950818 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051273 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051317 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051585 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051344 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051585 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051439 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa795690-3689-45c2-b455-a1b7b58e873d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051585 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051494 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqs5\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-kube-api-access-zdqs5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051585 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051527 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa795690-3689-45c2-b455-a1b7b58e873d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051585 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051551 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051886 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051584 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051886 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051620 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.051886 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.051872 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.052066 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.052010 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.052250 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.052223 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa795690-3689-45c2-b455-a1b7b58e873d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.052438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.052310 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.052558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.052531 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.053836 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.053814 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa795690-3689-45c2-b455-a1b7b58e873d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.054091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.054073 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa795690-3689-45c2-b455-a1b7b58e873d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.105424 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.105354 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.105568 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.105546 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqs5\" (UniqueName: \"kubernetes.io/projected/fa795690-3689-45c2-b455-a1b7b58e873d-kube-api-access-zdqs5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-24nqh\" (UID: \"fa795690-3689-45c2-b455-a1b7b58e873d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.164645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.164605 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:25.579015 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.578989 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh"] Nov 02 21:21:25.580942 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:21:25.580914 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa795690_3689_45c2_b455_a1b7b58e873d.slice/crio-c741d0b3c4b7d258089679509f0cf74a57b32538ca9ca863bc0938b079688cc5 WatchSource:0}: Error finding container c741d0b3c4b7d258089679509f0cf74a57b32538ca9ca863bc0938b079688cc5: Status 404 returned error can't find the container with id c741d0b3c4b7d258089679509f0cf74a57b32538ca9ca863bc0938b079688cc5 Nov 02 21:21:25.583340 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.583311 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:21:25.583426 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.583376 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:21:25.583426 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.583406 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:21:25.916556 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.916462 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" event={"ID":"fa795690-3689-45c2-b455-a1b7b58e873d","Type":"ContainerStarted","Data":"39dd793e35bea69472168a92d076eb5eeb5ffa7b5af4ef9097153d25736041cf"} Nov 02 21:21:25.916556 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:25.916547 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" event={"ID":"fa795690-3689-45c2-b455-a1b7b58e873d","Type":"ContainerStarted","Data":"c741d0b3c4b7d258089679509f0cf74a57b32538ca9ca863bc0938b079688cc5"} Nov 02 21:21:26.032754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:26.032219 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podStartSLOduration=2.032197708 podStartE2EDuration="2.032197708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:26.030695387 +0000 UTC m=+5569.637239888" watchObservedRunningTime="2026-11-02 21:21:26.032197708 +0000 UTC m=+5569.638742207" Nov 02 21:21:26.165635 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:26.165600 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:26.167236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:26.167170 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:26.167236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:26.167221 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:27.165888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:27.165849 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:27.166391 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:27.165903 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:28.165436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:28.165400 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:28.165628 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:28.165451 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:29.165430 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:29.165398 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:29.165854 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:29.165452 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:30.165655 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:30.165623 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:30.166112 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:30.165675 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:31.165014 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:31.164977 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:31.165210 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:31.165034 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:32.165166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:32.165125 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 21:21:32.165599 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:32.165180 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 21:21:33.973497 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:33.973466 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:33.974024 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:33.973779 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:33.974839 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:21:33.974817 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" Nov 02 21:21:51.003720 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:21:51.003646 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:22:10.211016 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.210984 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn"] Nov 02 21:22:10.214594 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.214577 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.258549 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.258515 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk"] Nov 02 21:22:10.262014 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.261999 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.280583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280554 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280591 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280612 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280701 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280814 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8rb\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-kube-api-access-rj8rb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280841 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280894 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.280964 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.280917 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.281129 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.281009 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.351805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.351766 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk"] Nov 02 21:22:10.382485 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382455 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8rb\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-kube-api-access-rj8rb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.382675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382491 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.382675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382513 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.382675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382587 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.382675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382615 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7346bd-7650-4820-a6e3-65ef8979ff89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.382675 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382659 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382711 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382801 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382832 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpx24\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-kube-api-access-kpx24\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382905 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382951 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.382978 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383041 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383091 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383125 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383155 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383186 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383209 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383249 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383359 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383835 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383437 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383835 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383702 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.383835 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.383805 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.385248 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.385225 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.385384 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.385368 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.406813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.406791 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn"] Nov 02 21:22:10.436509 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.436486 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.436914 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.436893 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8rb\" (UniqueName: \"kubernetes.io/projected/14f88a84-e518-4c2c-abbd-8bc07a283a4d-kube-api-access-rj8rb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z44pn\" (UID: \"14f88a84-e518-4c2c-abbd-8bc07a283a4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.484695 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484638 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484695 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484669 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx24\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-kube-api-access-kpx24\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484695 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484687 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484714 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484755 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484779 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484859 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484903 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.484958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.484934 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7346bd-7650-4820-a6e3-65ef8979ff89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.485300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.485201 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.485300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.485256 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.485401 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.485295 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.485401 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.485369 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.485651 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.485634 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7346bd-7650-4820-a6e3-65ef8979ff89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.487378 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.487357 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.487745 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.487691 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.528063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.528039 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:10.538111 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.538091 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.538520 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.538501 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx24\" (UniqueName: \"kubernetes.io/projected/3d7346bd-7650-4820-a6e3-65ef8979ff89-kube-api-access-kpx24\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nzffk\" (UID: \"3d7346bd-7650-4820-a6e3-65ef8979ff89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.573967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.573929 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:10.937292 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.937213 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn"] Nov 02 21:22:10.978242 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:22:10.978217 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f88a84_e518_4c2c_abbd_8bc07a283a4d.slice/crio-6b8c3d5dc0c98fc0a53f95a6b671da39096d3731d6c02648daa00ab6cc4731db WatchSource:0}: Error finding container 6b8c3d5dc0c98fc0a53f95a6b671da39096d3731d6c02648daa00ab6cc4731db: Status 404 returned error can't find the container with id 6b8c3d5dc0c98fc0a53f95a6b671da39096d3731d6c02648daa00ab6cc4731db Nov 02 21:22:10.980459 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.980428 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:10.980527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.980499 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:10.980574 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.980525 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:10.987636 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.987617 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk"] Nov 02 21:22:10.989485 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:22:10.989462 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7346bd_7650_4820_a6e3_65ef8979ff89.slice/crio-244a53f69c1c7f4e607d6badda780d6548e1317a06940738326dc3361caa2ebf WatchSource:0}: Error finding container 244a53f69c1c7f4e607d6badda780d6548e1317a06940738326dc3361caa2ebf: Status 404 returned error can't find the container with id 244a53f69c1c7f4e607d6badda780d6548e1317a06940738326dc3361caa2ebf Nov 02 21:22:10.991512 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.991485 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:10.991593 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.991540 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:10.991593 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:10.991571 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:22:11.134436 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.134398 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" event={"ID":"14f88a84-e518-4c2c-abbd-8bc07a283a4d","Type":"ContainerStarted","Data":"e6faa86fef9a55638c3425cfd3ee099bf6d1666edf8bf3c13d0485f89f9953f3"} Nov 02 21:22:11.134764 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.134709 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" event={"ID":"14f88a84-e518-4c2c-abbd-8bc07a283a4d","Type":"ContainerStarted","Data":"6b8c3d5dc0c98fc0a53f95a6b671da39096d3731d6c02648daa00ab6cc4731db"} Nov 02 21:22:11.137888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.137861 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" event={"ID":"3d7346bd-7650-4820-a6e3-65ef8979ff89","Type":"ContainerStarted","Data":"c0c9efb0f6e70e05672854ca4e99007969d8276779ccdc811022b3628fc5242d"} Nov 02 21:22:11.138050 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.137918 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" event={"ID":"3d7346bd-7650-4820-a6e3-65ef8979ff89","Type":"ContainerStarted","Data":"244a53f69c1c7f4e607d6badda780d6548e1317a06940738326dc3361caa2ebf"} Nov 02 21:22:11.244890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.244835 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podStartSLOduration=1.244821859 podStartE2EDuration="1.244821859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:11.244075862 +0000 UTC m=+5614.850620373" watchObservedRunningTime="2026-11-02 21:22:11.244821859 +0000 UTC m=+5614.851366356" Nov 02 21:22:11.355362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.355302 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podStartSLOduration=1.355283952 podStartE2EDuration="1.355283952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:11.353828921 +0000 UTC m=+5614.960373430" watchObservedRunningTime="2026-11-02 21:22:11.355283952 +0000 UTC m=+5614.961828495" Nov 02 21:22:11.529924 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.529825 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:11.574149 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:11.574119 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:12.536754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:12.535386 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:12.536754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:12.535448 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:12.575929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:12.575827 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:12.575929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:12.575878 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:13.529181 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:13.529141 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:13.529479 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:13.529449 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:13.575120 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:13.575083 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:13.575554 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:13.575532 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:14.532754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:14.532134 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:14.532754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:14.532199 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:14.574259 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:14.574224 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:14.574427 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:14.574273 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:15.529540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:15.529107 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:15.529540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:15.529170 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:15.574960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:15.574927 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:15.575239 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:15.575209 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:16.536420 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.535840 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:16.536420 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.535923 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:16.781072 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.781024 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:16.781367 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.781334 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:16.876829 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.876722 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:22:16.909956 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.909921 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:22:16.975573 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:16.914879 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.027975 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.027890 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-006730b5-kserve-self-signed-certs\"" Nov 02 21:22:17.074319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074211 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.074319 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074282 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.074608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074384 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.074608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074494 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.074608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074537 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqj6m\" (UniqueName: \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.074608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.074568 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176178 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176076 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176178 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176134 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqj6m\" (UniqueName: \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176178 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176162 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176257 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176294 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176501 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176319 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.176678 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.176634 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.177052 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.177018 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.177273 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.177250 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.179626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.179601 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.179921 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.179895 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.230493 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.230437 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqj6m\" (UniqueName: \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\") pod \"e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.263885 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.263851 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:22:17.267875 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.267842 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.332872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.320309 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-006730b5-epp-sa-dockercfg-452s6\"" Nov 02 21:22:17.332872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.328809 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:17.332872 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.332547 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:22:17.378922 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.378880 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckncv\" (UniqueName: \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.379099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.379076 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.480104 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.480061 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckncv\" (UniqueName: \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.480275 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.480225 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.482956 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.482931 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.528517 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.528484 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:17.528666 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.528541 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:17.534123 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.534090 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckncv\" (UniqueName: \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\") pod \"e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.575346 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.575314 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:17.575682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.575369 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:17.586908 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.586881 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:17.729613 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.729580 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:22:17.747246 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:22:17.747211 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1456f909_5d6d_4e27_97fa_62b09c0050fb.slice/crio-656ee5c8c1e7f5cefc5ab8b382c854c101c8634c46cc41bb19b0e6749ff6cf1f WatchSource:0}: Error finding container 656ee5c8c1e7f5cefc5ab8b382c854c101c8634c46cc41bb19b0e6749ff6cf1f: Status 404 returned error can't find the container with id 656ee5c8c1e7f5cefc5ab8b382c854c101c8634c46cc41bb19b0e6749ff6cf1f Nov 02 21:22:17.970582 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:17.970558 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:22:17.973112 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:22:17.973081 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59c2a04_6aba_4a2b_8c43_810ed6dbf9c6.slice/crio-ccb5c2ba5de7c912a7a6fdbbe967a0ff7696c8906ea4bb2155809b5821dc6a4d WatchSource:0}: Error finding container ccb5c2ba5de7c912a7a6fdbbe967a0ff7696c8906ea4bb2155809b5821dc6a4d: Status 404 returned error can't find the container with id ccb5c2ba5de7c912a7a6fdbbe967a0ff7696c8906ea4bb2155809b5821dc6a4d Nov 02 21:22:18.194925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.191927 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" event={"ID":"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6","Type":"ContainerStarted","Data":"ccb5c2ba5de7c912a7a6fdbbe967a0ff7696c8906ea4bb2155809b5821dc6a4d"} Nov 02 21:22:18.194925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.193942 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" event={"ID":"1456f909-5d6d-4e27-97fa-62b09c0050fb","Type":"ContainerStarted","Data":"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d"} Nov 02 21:22:18.194925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.193982 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" event={"ID":"1456f909-5d6d-4e27-97fa-62b09c0050fb","Type":"ContainerStarted","Data":"656ee5c8c1e7f5cefc5ab8b382c854c101c8634c46cc41bb19b0e6749ff6cf1f"} Nov 02 21:22:18.303555 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.303488 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" podStartSLOduration=2.303466057 podStartE2EDuration="2.303466057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:18.302117338 +0000 UTC m=+5621.908661839" watchObservedRunningTime="2026-11-02 21:22:18.303466057 +0000 UTC m=+5621.910010555" Nov 02 21:22:18.333304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.333265 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:18.528797 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.528761 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 21:22:18.528985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.528814 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 21:22:18.574478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.574431 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 21:22:18.574633 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:18.574498 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 21:22:20.529724 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:20.529680 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:20.530238 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:20.529756 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:20.617359 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:20.616894 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:20.617359 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:20.616957 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:21.014545 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:22:21.014469 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:22:21.570912 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:21.570867 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:21.571443 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:21.570943 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:21.658017 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:21.657979 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:21.658215 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:21.658031 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:22.102095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.102058 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:22.102718 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.102692 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:22.118757 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.118311 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" Nov 02 21:22:22.169537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.169507 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:22.169957 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.169930 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:22.171175 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:22.171149 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" Nov 02 21:22:23.234705 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:23.232686 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" event={"ID":"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6","Type":"ContainerStarted","Data":"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d"} Nov 02 21:22:23.234705 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:23.233820 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:22:27.332816 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:27.332784 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:27.345660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:27.345636 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:27.471867 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:27.471817 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podStartSLOduration=5.517528826 podStartE2EDuration="10.471803319s" totalImagesPullingTime="4.954274493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:23.343280958 +0000 UTC m=+5626.949825456" watchObservedRunningTime="2026-11-02 21:22:27.471803319 +0000 UTC m=+5631.078347814" Nov 02 21:22:28.301110 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:28.301078 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:22:51.004917 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:22:51.004876 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:22:54.264238 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:22:54.264209 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:23:08.159758 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:08.158847 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:08.159758 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:08.158951 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:08.224755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:08.220828 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:08.224755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:08.220892 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:21.003396 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:23:21.003317 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:23:29.156868 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:29.156836 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:23:29.163309 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:29.163278 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:23:34.264996 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:34.264952 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:23:44.177768 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.177695 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" containerID="cri-o://2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d" gracePeriod=45 Nov 02 21:23:44.182780 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.182741 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerName="main" containerID="cri-o://da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d" gracePeriod=45 Nov 02 21:23:44.263859 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.263817 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:23:44.291166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.291129 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.212:8000/ready\": dial tcp 10.133.0.212:8000: connect: connection refused" Nov 02 21:23:44.441047 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.440991 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:23:44.504778 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504744 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.504926 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504799 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.504926 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504906 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.505033 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504940 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.505033 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504971 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.505033 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.504991 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:23:44.505033 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505000 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\" (UniqueName: \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\") pod \"1456f909-5d6d-4e27-97fa-62b09c0050fb\" (UID: \"1456f909-5d6d-4e27-97fa-62b09c0050fb\") " Nov 02 21:23:44.505250 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505170 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:23:44.505250 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505201 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:23:44.505421 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505407 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.505469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505424 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.505469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.505433 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.507075 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.507048 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:23:44.507195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.507155 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "kube-api-access-jqj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:23:44.507195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.507164 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs" pod "1456f909-5d6d-4e27-97fa-62b09c0050fb" (UID: "1456f909-5d6d-4e27-97fa-62b09c0050fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:23:44.606638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.606612 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1456f909-5d6d-4e27-97fa-62b09c0050fb-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.606638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.606637 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1456f909-5d6d-4e27-97fa-62b09c0050fb-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.606638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.606647 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqj6m\" (UniqueName: \"kubernetes.io/projected/1456f909-5d6d-4e27-97fa-62b09c0050fb-kube-api-access-jqj6m\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:44.711132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.711103 5136 generic.go:356] "Generic (PLEG): container finished" podID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerID="da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d" exitCode=0 Nov 02 21:23:44.711304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.711139 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" event={"ID":"1456f909-5d6d-4e27-97fa-62b09c0050fb","Type":"ContainerDied","Data":"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d"} Nov 02 21:23:44.711304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.711167 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" event={"ID":"1456f909-5d6d-4e27-97fa-62b09c0050fb","Type":"ContainerDied","Data":"656ee5c8c1e7f5cefc5ab8b382c854c101c8634c46cc41bb19b0e6749ff6cf1f"} Nov 02 21:23:44.711304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.711142 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv" Nov 02 21:23:44.711304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.711183 5136 scope.go:122] "RemoveContainer" containerID="da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d" Nov 02 21:23:44.719628 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.719610 5136 scope.go:122] "RemoveContainer" containerID="da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d" Nov 02 21:23:44.720000 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:23:44.719977 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d\": container with ID starting with da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d not found: ID does not exist" containerID="da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d" Nov 02 21:23:44.720068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.720007 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d"} err="failed to get container status \"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d\": rpc error: code = NotFound desc = could not find container \"da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d\": container with ID starting with da3a44bab9e55de07c3c0123980f84d6207337b4586fa890c3bc697fe68a595d not found: ID does not exist" Nov 02 21:23:44.866798 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.866761 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:23:44.874286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:44.874250 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-fb7f4698-ds9wv"] Nov 02 21:23:46.545712 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:46.545673 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" path="/var/lib/kubelet/pods/1456f909-5d6d-4e27-97fa-62b09c0050fb/volumes" Nov 02 21:23:51.133502 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:23:51.133463 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:23:54.263975 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:54.263930 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:23:54.264381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:23:54.264042 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:24:04.264562 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:04.264524 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:24:14.262776 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:24:14.262750 5136 logging.go:55] [core] [Channel #8818 SubChannel #8819]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.213:9003", ServerName: "10.133.0.213:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.213:9003: connect: connection refused" Nov 02 21:24:14.433813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.433786 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:24:14.584679 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.584648 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\") pod \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " Nov 02 21:24:14.584869 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.584797 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\" (UniqueName: \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\") pod \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\" (UID: \"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6\") " Nov 02 21:24:14.586827 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.586789 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs" pod "a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" (UID: "a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:24:14.586967 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.586856 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv" pod "a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" (UID: "a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6"). InnerVolumeSpecName "kube-api-access-ckncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:24:14.685998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.685970 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:14.685998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.685993 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckncv\" (UniqueName: \"kubernetes.io/projected/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6-kube-api-access-ckncv\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:14.892322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.892239 5136 generic.go:356] "Generic (PLEG): container finished" podID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerID="2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d" exitCode=0 Nov 02 21:24:14.892500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.892314 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" Nov 02 21:24:14.892500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.892330 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" event={"ID":"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6","Type":"ContainerDied","Data":"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d"} Nov 02 21:24:14.892500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.892406 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" event={"ID":"a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6","Type":"ContainerDied","Data":"ccb5c2ba5de7c912a7a6fdbbe967a0ff7696c8906ea4bb2155809b5821dc6a4d"} Nov 02 21:24:14.892500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.892437 5136 scope.go:122] "RemoveContainer" containerID="2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d" Nov 02 21:24:14.904988 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.904965 5136 scope.go:122] "RemoveContainer" containerID="2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d" Nov 02 21:24:14.905366 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:24:14.905300 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d\": container with ID starting with 2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d not found: ID does not exist" containerID="2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d" Nov 02 21:24:14.905366 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:14.905336 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d"} err="failed to get container status \"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d\": rpc error: code = NotFound desc = could not find container \"2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d\": container with ID starting with 2bb50f7dbf55687026059a7a43ce2d745ac3eea7f85b2012ae216d32ca48b19d not found: ID does not exist" Nov 02 21:24:15.046810 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:15.046775 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:24:15.050954 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:15.050928 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8"] Nov 02 21:24:15.262789 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:15.262748 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-006730b5-kserve-router-scheduler-5cc6b5fb97-lkbd8" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.213:9003\" within 1s: context deadline exceeded" Nov 02 21:24:16.545851 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:16.545816 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" path="/var/lib/kubelet/pods/a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6/volumes" Nov 02 21:24:21.003196 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:24:21.003147 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:24:21.254846 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.254768 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:24:21.255181 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255169 5136 cpu_manager.go:414] "containerMap: removing container" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerName="main" Nov 02 21:24:21.255228 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255182 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerName="main" Nov 02 21:24:21.255228 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255204 5136 cpu_manager.go:414] "containerMap: removing container" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" Nov 02 21:24:21.255228 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255209 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" Nov 02 21:24:21.255330 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255273 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="1456f909-5d6d-4e27-97fa-62b09c0050fb" containerName="main" Nov 02 21:24:21.255330 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.255284 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59c2a04-6aba-4a2b-8c43-810ed6dbf9c6" containerName="main" Nov 02 21:24:21.259628 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.259609 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.356905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.356868 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-742833\"/\"openshift-service-ca.crt\"" Nov 02 21:24:21.356905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.356873 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-742833\"/\"default-dockercfg-l54tb\"" Nov 02 21:24:21.357121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.356903 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-742833\"/\"simulator-single-main-742833-kserve-self-signed-certs\"" Nov 02 21:24:21.357121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.357021 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-742833\"/\"kube-root-ca.crt\"" Nov 02 21:24:21.358555 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.358533 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:24:21.360507 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360483 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.360618 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360595 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.360689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360632 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv62h\" (UniqueName: \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.360768 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360710 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.360837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360774 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.360837 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.360829 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462335 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462297 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462348 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv62h\" (UniqueName: \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462377 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462415 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462442 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462538 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462524 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462877 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462836 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462940 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462899 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.462940 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.462917 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.464714 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.464686 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.464962 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.464946 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.516630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.516573 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv62h\" (UniqueName: \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\") pod \"simulator-single-main-742833-kserve-54bd65dc8f-kw4xc\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.571667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.571638 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:21.932937 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:21.932913 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:24:21.935063 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:24:21.935035 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc837a6c9_08de_4b36_8d89_7404e58daf7e.slice/crio-09bf044ac48f5fd551f83afdb1433d6cdd3f0586938252f3ed5e5198c0cdb023 WatchSource:0}: Error finding container 09bf044ac48f5fd551f83afdb1433d6cdd3f0586938252f3ed5e5198c0cdb023: Status 404 returned error can't find the container with id 09bf044ac48f5fd551f83afdb1433d6cdd3f0586938252f3ed5e5198c0cdb023 Nov 02 21:24:22.933853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:22.933814 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" event={"ID":"c837a6c9-08de-4b36-8d89-7404e58daf7e","Type":"ContainerStarted","Data":"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9"} Nov 02 21:24:22.933853 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:22.933857 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" event={"ID":"c837a6c9-08de-4b36-8d89-7404e58daf7e","Type":"ContainerStarted","Data":"09bf044ac48f5fd551f83afdb1433d6cdd3f0586938252f3ed5e5198c0cdb023"} Nov 02 21:24:23.043295 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:23.043244 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" podStartSLOduration=2.043228688 podStartE2EDuration="2.043228688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:23.04149883 +0000 UTC m=+5746.648043328" watchObservedRunningTime="2026-11-02 21:24:23.043228688 +0000 UTC m=+5746.649773185" Nov 02 21:24:23.572037 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:23.572006 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:31.572194 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:31.572157 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:31.584755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:31.584705 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:31.989903 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:24:31.989876 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:24:51.004369 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:24:51.004282 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:25:20.689907 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:20.689873 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:25:21.003761 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:25:21.003711 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:25:35.709969 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:35.709897 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" podUID="c837a6c9-08de-4b36-8d89-7404e58daf7e" containerName="main" containerID="cri-o://e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9" gracePeriod=45 Nov 02 21:25:35.960963 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:35.960904 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074649 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\" (UniqueName: \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074744 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074781 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074815 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074844 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.075565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.074914 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\") pod \"c837a6c9-08de-4b36-8d89-7404e58daf7e\" (UID: \"c837a6c9-08de-4b36-8d89-7404e58daf7e\") " Nov 02 21:25:36.076806 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.076477 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:25:36.076806 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.076588 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:25:36.076806 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.076795 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:25:36.079301 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.079245 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "kube-api-access-lv62h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:25:36.079469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.079289 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:25:36.079604 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.079571 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs" pod "c837a6c9-08de-4b36-8d89-7404e58daf7e" (UID: "c837a6c9-08de-4b36-8d89-7404e58daf7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176391 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv62h\" (UniqueName: \"kubernetes.io/projected/c837a6c9-08de-4b36-8d89-7404e58daf7e-kube-api-access-lv62h\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176423 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176440 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837a6c9-08de-4b36-8d89-7404e58daf7e-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176452 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176464 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.176478 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.176475 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c837a6c9-08de-4b36-8d89-7404e58daf7e-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:36.276176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.276083 5136 generic.go:356] "Generic (PLEG): container finished" podID="c837a6c9-08de-4b36-8d89-7404e58daf7e" containerID="e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9" exitCode=0 Nov 02 21:25:36.276176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.276135 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" Nov 02 21:25:36.276176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.276168 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" event={"ID":"c837a6c9-08de-4b36-8d89-7404e58daf7e","Type":"ContainerDied","Data":"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9"} Nov 02 21:25:36.276422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.276210 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc" event={"ID":"c837a6c9-08de-4b36-8d89-7404e58daf7e","Type":"ContainerDied","Data":"09bf044ac48f5fd551f83afdb1433d6cdd3f0586938252f3ed5e5198c0cdb023"} Nov 02 21:25:36.276422 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.276229 5136 scope.go:122] "RemoveContainer" containerID="e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9" Nov 02 21:25:36.285931 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.285914 5136 scope.go:122] "RemoveContainer" containerID="e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9" Nov 02 21:25:36.286197 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:25:36.286176 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9\": container with ID starting with e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9 not found: ID does not exist" containerID="e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9" Nov 02 21:25:36.286270 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.286207 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9"} err="failed to get container status \"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9\": rpc error: code = NotFound desc = could not find container \"e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9\": container with ID starting with e565f0bcc537d17e461965bb3e7e96ad9e06333d06d14b68e70410fba2dee7e9 not found: ID does not exist" Nov 02 21:25:36.430829 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.430795 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:25:36.478069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.478039 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-742833/simulator-single-main-742833-kserve-54bd65dc8f-kw4xc"] Nov 02 21:25:36.546021 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:25:36.545949 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c837a6c9-08de-4b36-8d89-7404e58daf7e" path="/var/lib/kubelet/pods/c837a6c9-08de-4b36-8d89-7404e58daf7e/volumes" Nov 02 21:25:51.002632 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:25:51.002595 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:26:18.359328 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.359234 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:26:18.359937 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.359920 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c837a6c9-08de-4b36-8d89-7404e58daf7e" containerName="main" Nov 02 21:26:18.360066 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.359943 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c837a6c9-08de-4b36-8d89-7404e58daf7e" containerName="main" Nov 02 21:26:18.360066 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.360051 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c837a6c9-08de-4b36-8d89-7404e58daf7e" containerName="main" Nov 02 21:26:18.363245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.363224 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.415804 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.415773 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:26:18.460994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.460965 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f41fab3d-kserve-self-signed-certs\"" Nov 02 21:26:18.486121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486090 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.486260 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486211 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flw4t\" (UniqueName: \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.486306 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486263 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.486306 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486282 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.486306 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486301 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.486434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.486345 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587045 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587016 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flw4t\" (UniqueName: \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587070 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587086 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587101 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587118 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587230 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587162 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587579 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587607 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.587689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.587637 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.589639 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.589617 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.589885 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.589870 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.640950 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.640887 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flw4t\" (UniqueName: \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\") pod \"e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:18.674588 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:18.674561 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:19.047091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:19.047067 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:26:19.049204 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:26:19.049179 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce7d1f23_5460_4cf4_b291_ae934b5372a2.slice/crio-260ac2c26b913c672773fb8529f886d14c681497d441a1cbf178b07a9d8abd7c WatchSource:0}: Error finding container 260ac2c26b913c672773fb8529f886d14c681497d441a1cbf178b07a9d8abd7c: Status 404 returned error can't find the container with id 260ac2c26b913c672773fb8529f886d14c681497d441a1cbf178b07a9d8abd7c Nov 02 21:26:19.470251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:19.470219 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" event={"ID":"ce7d1f23-5460-4cf4-b291-ae934b5372a2","Type":"ContainerStarted","Data":"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5"} Nov 02 21:26:19.470781 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:19.470259 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" event={"ID":"ce7d1f23-5460-4cf4-b291-ae934b5372a2","Type":"ContainerStarted","Data":"260ac2c26b913c672773fb8529f886d14c681497d441a1cbf178b07a9d8abd7c"} Nov 02 21:26:19.580655 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:19.580600 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" podStartSLOduration=1.5805841809999999 podStartE2EDuration="1.580584181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:19.579267563 +0000 UTC m=+5863.185812060" watchObservedRunningTime="2026-11-02 21:26:19.580584181 +0000 UTC m=+5863.187128681" Nov 02 21:26:19.675198 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:19.675166 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:21.002567 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:26:21.002529 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:26:25.643754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:25.642838 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:25.643754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:25.642899 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:28.674994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:28.674959 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:28.687317 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:28.687294 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:29.542406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:29.542372 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:26:45.291594 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.291556 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:26:45.295207 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.295187 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389072 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389040 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389105 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389236 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389196 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389255 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389285 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcnw\" (UniqueName: \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.389399 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.389361 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.393415 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.393398 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f41fab3d-kserve-self-signed-certs\"" Nov 02 21:26:45.395017 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.394997 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:26:45.490481 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490448 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.490686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490485 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcnw\" (UniqueName: \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.490686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490542 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.490686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490593 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.490686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490679 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.490961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490706 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.491027 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.490987 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.491084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.491022 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.491084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.491058 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.492887 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.492866 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.493412 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.493388 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.544512 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.544455 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcnw\" (UniqueName: \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\") pod \"e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.606984 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.606961 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:45.974780 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:45.974753 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:26:45.976520 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:26:45.976492 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf005406_da82_482c_8a91_31706222137c.slice/crio-3ba530237b605eff4c04054cd09d66eebeb422095aeb068da6570864a7148efd WatchSource:0}: Error finding container 3ba530237b605eff4c04054cd09d66eebeb422095aeb068da6570864a7148efd: Status 404 returned error can't find the container with id 3ba530237b605eff4c04054cd09d66eebeb422095aeb068da6570864a7148efd Nov 02 21:26:46.609626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:46.609590 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" event={"ID":"bf005406-da82-482c-8a91-31706222137c","Type":"ContainerStarted","Data":"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297"} Nov 02 21:26:46.609626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:46.609634 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" event={"ID":"bf005406-da82-482c-8a91-31706222137c","Type":"ContainerStarted","Data":"3ba530237b605eff4c04054cd09d66eebeb422095aeb068da6570864a7148efd"} Nov 02 21:26:46.718402 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:46.718347 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" podStartSLOduration=1.71833365 podStartE2EDuration="1.71833365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:46.716911311 +0000 UTC m=+5890.323455810" watchObservedRunningTime="2026-11-02 21:26:46.71833365 +0000 UTC m=+5890.324878139" Nov 02 21:26:47.607905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:47.607870 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:51.017213 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:26:51.017174 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:26:53.150141 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:53.150102 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:53.150800 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:53.150764 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:53.217998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:53.217853 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:53.217998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:53.217931 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:55.607553 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:55.607520 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:55.619934 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:55.619909 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:26:55.703286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:26:55.703259 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:27:21.003327 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:27:21.003293 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:27:32.593606 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:32.593572 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:27:35.339960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:35.339932 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:27:47.613971 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.613858 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" podUID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" containerName="main" containerID="cri-o://7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5" gracePeriod=45 Nov 02 21:27:47.854885 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.854862 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:27:47.982682 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982649 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.982863 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982691 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.982863 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982723 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\" (UniqueName: \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.982863 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982762 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.983060 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982868 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.983060 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.982920 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\") pod \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\" (UID: \"ce7d1f23-5460-4cf4-b291-ae934b5372a2\") " Nov 02 21:27:47.983060 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983038 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:47.983226 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983096 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:47.983324 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983299 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:47.983451 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983429 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:47.983591 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983454 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:47.983591 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.983494 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:47.985030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.985000 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:47.985132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.985022 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:27:47.985132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:47.985062 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t" pod "ce7d1f23-5460-4cf4-b291-ae934b5372a2" (UID: "ce7d1f23-5460-4cf4-b291-ae934b5372a2"). InnerVolumeSpecName "kube-api-access-flw4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:27:48.000183 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.000162 5136 generic.go:356] "Generic (PLEG): container finished" podID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" containerID="7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5" exitCode=0 Nov 02 21:27:48.000312 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.000217 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" Nov 02 21:27:48.000312 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.000249 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" event={"ID":"ce7d1f23-5460-4cf4-b291-ae934b5372a2","Type":"ContainerDied","Data":"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5"} Nov 02 21:27:48.000312 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.000290 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4" event={"ID":"ce7d1f23-5460-4cf4-b291-ae934b5372a2","Type":"ContainerDied","Data":"260ac2c26b913c672773fb8529f886d14c681497d441a1cbf178b07a9d8abd7c"} Nov 02 21:27:48.000312 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.000307 5136 scope.go:122] "RemoveContainer" containerID="7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5" Nov 02 21:27:48.009341 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.009322 5136 scope.go:122] "RemoveContainer" containerID="7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5" Nov 02 21:27:48.009607 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:27:48.009588 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5\": container with ID starting with 7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5 not found: ID does not exist" containerID="7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5" Nov 02 21:27:48.009680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.009614 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5"} err="failed to get container status \"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5\": rpc error: code = NotFound desc = could not find container \"7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5\": container with ID starting with 7821bf323d7922b1f821651b770e9bfb5ab364c1522037a4ea47d0a137d451a5 not found: ID does not exist" Nov 02 21:27:48.084923 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.084903 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7d1f23-5460-4cf4-b291-ae934b5372a2-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:48.084923 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.084925 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flw4t\" (UniqueName: \"kubernetes.io/projected/ce7d1f23-5460-4cf4-b291-ae934b5372a2-kube-api-access-flw4t\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:48.085063 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.084936 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7d1f23-5460-4cf4-b291-ae934b5372a2-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:48.156243 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.156205 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:27:48.160991 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.160967 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f41fab3d-kserve-7f4bb4bb7c-ng7z4"] Nov 02 21:27:48.544928 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:48.544893 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" path="/var/lib/kubelet/pods/ce7d1f23-5460-4cf4-b291-ae934b5372a2/volumes" Nov 02 21:27:50.359743 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.359674 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" podUID="bf005406-da82-482c-8a91-31706222137c" containerName="main" containerID="cri-o://f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297" gracePeriod=45 Nov 02 21:27:50.613426 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.613367 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:27:50.811473 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811443 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\" (UniqueName: \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811633 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811516 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811633 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811612 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811757 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811659 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811757 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811688 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811841 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811753 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\") pod \"bf005406-da82-482c-8a91-31706222137c\" (UID: \"bf005406-da82-482c-8a91-31706222137c\") " Nov 02 21:27:50.811841 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811802 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:50.811956 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.811926 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:50.812132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.812100 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:50.812132 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.812119 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:50.812285 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.812144 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:50.813911 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.813887 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:27:50.814286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.814254 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:27:50.814286 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.814258 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw" pod "bf005406-da82-482c-8a91-31706222137c" (UID: "bf005406-da82-482c-8a91-31706222137c"). InnerVolumeSpecName "kube-api-access-4qcnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:27:50.912881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.912803 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf005406-da82-482c-8a91-31706222137c-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:50.912881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.912829 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:50.912881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.912838 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf005406-da82-482c-8a91-31706222137c-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:50.912881 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:50.912849 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qcnw\" (UniqueName: \"kubernetes.io/projected/bf005406-da82-482c-8a91-31706222137c-kube-api-access-4qcnw\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:51.003721 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:27:51.003687 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:27:51.015873 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.015849 5136 generic.go:356] "Generic (PLEG): container finished" podID="bf005406-da82-482c-8a91-31706222137c" containerID="f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297" exitCode=0 Nov 02 21:27:51.016019 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.015895 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" Nov 02 21:27:51.016019 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.015928 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" event={"ID":"bf005406-da82-482c-8a91-31706222137c","Type":"ContainerDied","Data":"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297"} Nov 02 21:27:51.016019 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.015970 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f" event={"ID":"bf005406-da82-482c-8a91-31706222137c","Type":"ContainerDied","Data":"3ba530237b605eff4c04054cd09d66eebeb422095aeb068da6570864a7148efd"} Nov 02 21:27:51.016019 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.015986 5136 scope.go:122] "RemoveContainer" containerID="f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297" Nov 02 21:27:51.024718 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.024702 5136 scope.go:122] "RemoveContainer" containerID="f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297" Nov 02 21:27:51.025001 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:27:51.024983 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297\": container with ID starting with f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297 not found: ID does not exist" containerID="f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297" Nov 02 21:27:51.025087 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.025008 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297"} err="failed to get container status \"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297\": rpc error: code = NotFound desc = could not find container \"f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297\": container with ID starting with f1c3bcc4f2081929099ef17fdfd47544f1022a64c51d8579c1b3960c1c2c6297 not found: ID does not exist" Nov 02 21:27:51.169579 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.169496 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:27:51.216864 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:51.216828 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f41fab3d-kserve-6d548bd769-p867f"] Nov 02 21:27:52.544870 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:52.544842 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf005406-da82-482c-8a91-31706222137c" path="/var/lib/kubelet/pods/bf005406-da82-482c-8a91-31706222137c/volumes" Nov 02 21:27:55.641382 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:55.640860 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:27:55.641382 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:27:55.640934 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:28:18.518320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518273 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:28:18.518755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518723 5136 cpu_manager.go:414] "containerMap: removing container" podUID="bf005406-da82-482c-8a91-31706222137c" containerName="main" Nov 02 21:28:18.518755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518752 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf005406-da82-482c-8a91-31706222137c" containerName="main" Nov 02 21:28:18.518838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518775 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" containerName="main" Nov 02 21:28:18.518838 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518781 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" containerName="main" Nov 02 21:28:18.518903 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518882 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf005406-da82-482c-8a91-31706222137c" containerName="main" Nov 02 21:28:18.518903 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.518891 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7d1f23-5460-4cf4-b291-ae934b5372a2" containerName="main" Nov 02 21:28:18.521706 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.521686 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.573901 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.573872 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:28:18.688608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.688567 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lwd\" (UniqueName: \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.688608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.688611 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.789990 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.789904 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lwd\" (UniqueName: \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.789990 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.789949 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.790662 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.790637 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:18.843598 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:18.843566 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lwd\" (UniqueName: \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\") pod \"payload-pre-processing-86845cc5b4-tnh9w\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:19.134888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:19.134799 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:19.499630 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:19.499605 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:28:19.502442 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:28:19.502412 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5093a7_52ec_4a9c_be73_8b404c4ea48f.slice/crio-009fcb6768f5447812f74cb9303710ba972f6ee2069437b9a4238393f5cf0cc1 WatchSource:0}: Error finding container 009fcb6768f5447812f74cb9303710ba972f6ee2069437b9a4238393f5cf0cc1: Status 404 returned error can't find the container with id 009fcb6768f5447812f74cb9303710ba972f6ee2069437b9a4238393f5cf0cc1 Nov 02 21:28:20.186774 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:20.186724 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" event={"ID":"5a5093a7-52ec-4a9c-be73-8b404c4ea48f","Type":"ContainerStarted","Data":"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc"} Nov 02 21:28:20.187212 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:20.186812 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:20.187212 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:20.186838 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" event={"ID":"5a5093a7-52ec-4a9c-be73-8b404c4ea48f","Type":"ContainerStarted","Data":"009fcb6768f5447812f74cb9303710ba972f6ee2069437b9a4238393f5cf0cc1"} Nov 02 21:28:20.295229 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:20.295176 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" podStartSLOduration=2.295161768 podStartE2EDuration="2.295161768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:20.293477393 +0000 UTC m=+5983.900021904" watchObservedRunningTime="2026-11-02 21:28:20.295161768 +0000 UTC m=+5983.901706263" Nov 02 21:28:21.003628 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:28:21.003593 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:28:31.194772 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:31.194715 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:28:33.189552 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.189516 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:28:33.193380 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.193350 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.233012 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.232987 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.233172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.233032 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpzb\" (UniqueName: \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.246890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.246854 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:28:33.334534 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.334501 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.334703 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.334547 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpzb\" (UniqueName: \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.335248 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.335223 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.389002 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.388970 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpzb\" (UniqueName: \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\") pod \"payload-processing-68fc5b85c8-4q2r9\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.506683 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.506652 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:33.867925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:33.867901 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:28:33.869403 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:28:33.869377 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0477d924_27c7_43fe_beb6_e3d9936b8c0d.slice/crio-702977755dc28278f0b30c978c32405501ac242af55ffb4a1a62825710026332 WatchSource:0}: Error finding container 702977755dc28278f0b30c978c32405501ac242af55ffb4a1a62825710026332: Status 404 returned error can't find the container with id 702977755dc28278f0b30c978c32405501ac242af55ffb4a1a62825710026332 Nov 02 21:28:34.269583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:34.269549 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" event={"ID":"0477d924-27c7-43fe-beb6-e3d9936b8c0d","Type":"ContainerStarted","Data":"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013"} Nov 02 21:28:34.270069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:34.269636 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:34.270069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:34.269656 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" event={"ID":"0477d924-27c7-43fe-beb6-e3d9936b8c0d","Type":"ContainerStarted","Data":"702977755dc28278f0b30c978c32405501ac242af55ffb4a1a62825710026332"} Nov 02 21:28:34.377039 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:34.376990 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" podStartSLOduration=1.3769778 podStartE2EDuration="1.3769778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:34.376928089 +0000 UTC m=+5997.983472589" watchObservedRunningTime="2026-11-02 21:28:34.3769778 +0000 UTC m=+5997.983522298" Nov 02 21:28:45.278294 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:45.278261 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:28:45.435178 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:45.435148 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 21:28:47.443059 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.443026 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7f8f4d5bdc-f56nj"] Nov 02 21:28:47.448175 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.448155 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.498487 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.498460 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7f8f4d5bdc-f56nj"] Nov 02 21:28:47.568338 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.568309 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4rm\" (UniqueName: \"kubernetes.io/projected/6c777b53-481c-40bf-a1c2-d2c188988730-kube-api-access-6s4rm\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.568499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.568342 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c777b53-481c-40bf-a1c2-d2c188988730-plugins-config-volume\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.603657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.603628 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-668545c68-6zrh6"] Nov 02 21:28:47.607196 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.607178 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.659196 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.659170 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-668545c68-6zrh6"] Nov 02 21:28:47.669793 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.669766 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7nt\" (UniqueName: \"kubernetes.io/projected/04992037-54d2-409a-9364-9d1acab24b15-kube-api-access-dv7nt\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.669939 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.669804 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04992037-54d2-409a-9364-9d1acab24b15-plugins-config-volume\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.670039 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.670013 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4rm\" (UniqueName: \"kubernetes.io/projected/6c777b53-481c-40bf-a1c2-d2c188988730-kube-api-access-6s4rm\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.670182 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.670047 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c777b53-481c-40bf-a1c2-d2c188988730-plugins-config-volume\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.670676 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.670659 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c777b53-481c-40bf-a1c2-d2c188988730-plugins-config-volume\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.723842 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.723815 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4rm\" (UniqueName: \"kubernetes.io/projected/6c777b53-481c-40bf-a1c2-d2c188988730-kube-api-access-6s4rm\") pod \"payload-processing-7f8f4d5bdc-f56nj\" (UID: \"6c777b53-481c-40bf-a1c2-d2c188988730\") " pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.761170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.761145 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:47.771592 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.771567 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7nt\" (UniqueName: \"kubernetes.io/projected/04992037-54d2-409a-9364-9d1acab24b15-kube-api-access-dv7nt\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.771715 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.771625 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04992037-54d2-409a-9364-9d1acab24b15-plugins-config-volume\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.772608 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.772580 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04992037-54d2-409a-9364-9d1acab24b15-plugins-config-volume\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.826689 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.826628 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7nt\" (UniqueName: \"kubernetes.io/projected/04992037-54d2-409a-9364-9d1acab24b15-kube-api-access-dv7nt\") pod \"payload-pre-processing-668545c68-6zrh6\" (UID: \"04992037-54d2-409a-9364-9d1acab24b15\") " pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:47.920141 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:47.920106 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:48.125004 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.124979 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7f8f4d5bdc-f56nj"] Nov 02 21:28:48.126941 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:28:48.126915 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c777b53_481c_40bf_a1c2_d2c188988730.slice/crio-0c28e5ba95b89dafdeedd8833269b6e8fff8be110f49c43a3d7bc550e44a314c WatchSource:0}: Error finding container 0c28e5ba95b89dafdeedd8833269b6e8fff8be110f49c43a3d7bc550e44a314c: Status 404 returned error can't find the container with id 0c28e5ba95b89dafdeedd8833269b6e8fff8be110f49c43a3d7bc550e44a314c Nov 02 21:28:48.283290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.283265 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-668545c68-6zrh6"] Nov 02 21:28:48.284713 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:28:48.284688 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04992037_54d2_409a_9364_9d1acab24b15.slice/crio-403187b49fe42ef02113929d6d7e8731a91588024e9c4d0def6ee9364730ca81 WatchSource:0}: Error finding container 403187b49fe42ef02113929d6d7e8731a91588024e9c4d0def6ee9364730ca81: Status 404 returned error can't find the container with id 403187b49fe42ef02113929d6d7e8731a91588024e9c4d0def6ee9364730ca81 Nov 02 21:28:48.336484 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.336459 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" event={"ID":"04992037-54d2-409a-9364-9d1acab24b15","Type":"ContainerStarted","Data":"403187b49fe42ef02113929d6d7e8731a91588024e9c4d0def6ee9364730ca81"} Nov 02 21:28:48.337998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.337976 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" event={"ID":"6c777b53-481c-40bf-a1c2-d2c188988730","Type":"ContainerStarted","Data":"0343c4a3158532f0bffdf7f07138062891ab989d1c9327e60b78ab4bf23977ce"} Nov 02 21:28:48.338118 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.338014 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" event={"ID":"6c777b53-481c-40bf-a1c2-d2c188988730","Type":"ContainerStarted","Data":"0c28e5ba95b89dafdeedd8833269b6e8fff8be110f49c43a3d7bc550e44a314c"} Nov 02 21:28:48.338184 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.338111 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:48.445183 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:48.445129 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" podStartSLOduration=1.445114668 podStartE2EDuration="1.445114668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:48.444888023 +0000 UTC m=+6012.051432521" watchObservedRunningTime="2026-11-02 21:28:48.445114668 +0000 UTC m=+6012.051659162" Nov 02 21:28:49.344550 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:49.344513 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" event={"ID":"04992037-54d2-409a-9364-9d1acab24b15","Type":"ContainerStarted","Data":"f44348182e47d7c7fa470fc0aa58effbfe8f7ccf8187bd86ca28b6183b6f6787"} Nov 02 21:28:49.344754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:49.344604 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:28:49.452558 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:49.452505 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" podStartSLOduration=2.452492791 podStartE2EDuration="2.452492791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:49.45128447 +0000 UTC m=+6013.057828969" watchObservedRunningTime="2026-11-02 21:28:49.452492791 +0000 UTC m=+6013.059037289" Nov 02 21:28:50.435446 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.435384 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" podUID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" containerName="payload-processing" containerID="cri-o://13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd" gracePeriod=25 Nov 02 21:28:50.681352 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.681327 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 21:28:50.800483 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.800442 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\" (UniqueName: \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\") pod \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " Nov 02 21:28:50.800690 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.800534 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\") pod \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\" (UID: \"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f\") " Nov 02 21:28:50.801010 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.800981 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume" pod "fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" (UID: "fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:50.802792 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.802707 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67" pod "fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" (UID: "fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f"). InnerVolumeSpecName "kube-api-access-dxk67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:50.901982 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.901952 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:50.901982 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:50.901979 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxk67\" (UniqueName: \"kubernetes.io/projected/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f-kube-api-access-dxk67\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:51.003013 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:28:51.002981 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:28:51.354626 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.354594 5136 generic.go:356] "Generic (PLEG): container finished" podID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" containerID="13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd" exitCode=0 Nov 02 21:28:51.354862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.354639 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" Nov 02 21:28:51.354862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.354681 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" event={"ID":"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f","Type":"ContainerDied","Data":"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd"} Nov 02 21:28:51.354862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.354759 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vzlpf" event={"ID":"fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f","Type":"ContainerDied","Data":"4be1bb8cc2fcb85c0afd633102721d8424dc0cab54ba08e2927a2b60aa10a1ed"} Nov 02 21:28:51.354862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.354791 5136 scope.go:122] "RemoveContainer" containerID="13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd" Nov 02 21:28:51.363907 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.363826 5136 scope.go:122] "RemoveContainer" containerID="13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd" Nov 02 21:28:51.364085 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:28:51.364067 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd\": container with ID starting with 13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd not found: ID does not exist" containerID="13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd" Nov 02 21:28:51.364164 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.364096 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd"} err="failed to get container status \"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd\": rpc error: code = NotFound desc = could not find container \"13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd\": container with ID starting with 13c6a9732929be7a65c2b3f5d4a4e35a34f16fe3f2c3703fd7ad943c81ea42cd not found: ID does not exist" Nov 02 21:28:51.508907 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.508877 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 21:28:51.556334 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:51.556307 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vzlpf"] Nov 02 21:28:52.544624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:52.544591 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" path="/var/lib/kubelet/pods/fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f/volumes" Nov 02 21:28:59.347577 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:59.347525 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7f8f4d5bdc-f56nj" Nov 02 21:28:59.505053 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:28:59.505018 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:29:00.352276 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:00.352248 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-668545c68-6zrh6" Nov 02 21:29:00.508392 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:00.508364 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:29:04.505562 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.505517 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" podUID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" containerName="payload-processing" containerID="cri-o://f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013" gracePeriod=25 Nov 02 21:29:04.756530 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.756465 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:29:04.838343 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.838313 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\" (UniqueName: \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\") pod \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " Nov 02 21:29:04.838519 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.838349 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\") pod \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\" (UID: \"0477d924-27c7-43fe-beb6-e3d9936b8c0d\") " Nov 02 21:29:04.838796 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.838771 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume" pod "0477d924-27c7-43fe-beb6-e3d9936b8c0d" (UID: "0477d924-27c7-43fe-beb6-e3d9936b8c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:04.840388 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.840362 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb" pod "0477d924-27c7-43fe-beb6-e3d9936b8c0d" (UID: "0477d924-27c7-43fe-beb6-e3d9936b8c0d"). InnerVolumeSpecName "kube-api-access-rdpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:04.940068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.940037 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpzb\" (UniqueName: \"kubernetes.io/projected/0477d924-27c7-43fe-beb6-e3d9936b8c0d-kube-api-access-rdpzb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:04.940068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:04.940062 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0477d924-27c7-43fe-beb6-e3d9936b8c0d-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:05.422119 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.422078 5136 generic.go:356] "Generic (PLEG): container finished" podID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" containerID="f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013" exitCode=0 Nov 02 21:29:05.422322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.422132 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" Nov 02 21:29:05.422322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.422166 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" event={"ID":"0477d924-27c7-43fe-beb6-e3d9936b8c0d","Type":"ContainerDied","Data":"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013"} Nov 02 21:29:05.422322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.422227 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68fc5b85c8-4q2r9" event={"ID":"0477d924-27c7-43fe-beb6-e3d9936b8c0d","Type":"ContainerDied","Data":"702977755dc28278f0b30c978c32405501ac242af55ffb4a1a62825710026332"} Nov 02 21:29:05.422322 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.422250 5136 scope.go:122] "RemoveContainer" containerID="f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013" Nov 02 21:29:05.431659 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.431639 5136 scope.go:122] "RemoveContainer" containerID="f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013" Nov 02 21:29:05.432000 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:29:05.431980 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013\": container with ID starting with f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013 not found: ID does not exist" containerID="f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013" Nov 02 21:29:05.432103 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.432009 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013"} err="failed to get container status \"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013\": rpc error: code = NotFound desc = could not find container \"f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013\": container with ID starting with f188bf3d7e83be96bc3e1ffc96d9fd4bd01048fb6c9085ce5541557a005a7013 not found: ID does not exist" Nov 02 21:29:05.509158 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.509129 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" podUID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" containerName="payload-pre-processing" containerID="cri-o://cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc" gracePeriod=25 Nov 02 21:29:05.577003 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.576974 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:29:05.578453 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.578433 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-68fc5b85c8-4q2r9"] Nov 02 21:29:05.765517 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.765483 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:29:05.848432 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.848403 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\" (UniqueName: \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\") pod \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " Nov 02 21:29:05.848587 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.848525 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\") pod \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\" (UID: \"5a5093a7-52ec-4a9c-be73-8b404c4ea48f\") " Nov 02 21:29:05.848917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.848892 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume" pod "5a5093a7-52ec-4a9c-be73-8b404c4ea48f" (UID: "5a5093a7-52ec-4a9c-be73-8b404c4ea48f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:05.850495 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.850476 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd" pod "5a5093a7-52ec-4a9c-be73-8b404c4ea48f" (UID: "5a5093a7-52ec-4a9c-be73-8b404c4ea48f"). InnerVolumeSpecName "kube-api-access-q9lwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:05.950099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.950066 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:05.950099 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:05.950102 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lwd\" (UniqueName: \"kubernetes.io/projected/5a5093a7-52ec-4a9c-be73-8b404c4ea48f-kube-api-access-q9lwd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:06.299769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.299677 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:29:06.300147 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300131 5136 cpu_manager.go:414] "containerMap: removing container" podUID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" containerName="payload-processing" Nov 02 21:29:06.300206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300147 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" containerName="payload-processing" Nov 02 21:29:06.300206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300185 5136 cpu_manager.go:414] "containerMap: removing container" podUID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" containerName="payload-processing" Nov 02 21:29:06.300206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300191 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" containerName="payload-processing" Nov 02 21:29:06.300206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300199 5136 cpu_manager.go:414] "containerMap: removing container" podUID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" containerName="payload-pre-processing" Nov 02 21:29:06.300206 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300204 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" containerName="payload-pre-processing" Nov 02 21:29:06.300387 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300265 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe75c77e-1b1d-4489-9a57-d8d6d1fffc7f" containerName="payload-processing" Nov 02 21:29:06.300387 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300274 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" containerName="payload-pre-processing" Nov 02 21:29:06.300387 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.300281 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" containerName="payload-processing" Nov 02 21:29:06.303460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.303443 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.353783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.353756 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 21:29:06.353959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.353888 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 21:29:06.353959 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.353942 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 21:29:06.355861 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.355837 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:29:06.387315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.387257 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 21:29:06.387315 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.387297 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 21:29:06.401089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.401070 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v25gx\"" Nov 02 21:29:06.428098 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.428073 5136 generic.go:356] "Generic (PLEG): container finished" podID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" containerID="cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc" exitCode=0 Nov 02 21:29:06.428225 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.428124 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" Nov 02 21:29:06.428225 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.428150 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" event={"ID":"5a5093a7-52ec-4a9c-be73-8b404c4ea48f","Type":"ContainerDied","Data":"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc"} Nov 02 21:29:06.428225 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.428194 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w" event={"ID":"5a5093a7-52ec-4a9c-be73-8b404c4ea48f","Type":"ContainerDied","Data":"009fcb6768f5447812f74cb9303710ba972f6ee2069437b9a4238393f5cf0cc1"} Nov 02 21:29:06.428225 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.428216 5136 scope.go:122] "RemoveContainer" containerID="cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc" Nov 02 21:29:06.437325 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.437311 5136 scope.go:122] "RemoveContainer" containerID="cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc" Nov 02 21:29:06.437581 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:29:06.437563 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc\": container with ID starting with cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc not found: ID does not exist" containerID="cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc" Nov 02 21:29:06.437632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.437589 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc"} err="failed to get container status \"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc\": rpc error: code = NotFound desc = could not find container \"cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc\": container with ID starting with cc71c5166386ab2fe361bc9cda34ec97f5902df587fb057dce318ccedd8921dc not found: ID does not exist" Nov 02 21:29:06.456006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.455984 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.456185 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.456023 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.456185 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.456104 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb748\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.456266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.456230 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.545821 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.545791 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0477d924-27c7-43fe-beb6-e3d9936b8c0d" path="/var/lib/kubelet/pods/0477d924-27c7-43fe-beb6-e3d9936b8c0d/volumes" Nov 02 21:29:06.557557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.557501 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.557557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.557537 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.557557 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.557558 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.557791 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.557579 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb748\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.558157 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.558135 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.558336 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.558316 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.559985 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.559965 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.583752 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.583697 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:29:06.587156 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.587132 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-86845cc5b4-tnh9w"] Nov 02 21:29:06.611534 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.611507 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb748\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\") pod \"payload-processing-external-model-5bcd7997-srjs9\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:06.617526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:06.617358 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:07.013042 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:07.013004 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:29:07.019334 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:29:07.019305 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba7f808_53a0_43c4_ab6d_fc87c6b3d12e.slice/crio-d1f6c5ebdeb792d347139c9c0c66029cb86d9e55a54e0ab0420015b65ae048f7 WatchSource:0}: Error finding container d1f6c5ebdeb792d347139c9c0c66029cb86d9e55a54e0ab0420015b65ae048f7: Status 404 returned error can't find the container with id d1f6c5ebdeb792d347139c9c0c66029cb86d9e55a54e0ab0420015b65ae048f7 Nov 02 21:29:07.435893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:07.435810 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" event={"ID":"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e","Type":"ContainerStarted","Data":"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16"} Nov 02 21:29:07.435893 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:07.435862 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" event={"ID":"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e","Type":"ContainerStarted","Data":"d1f6c5ebdeb792d347139c9c0c66029cb86d9e55a54e0ab0420015b65ae048f7"} Nov 02 21:29:07.436125 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:07.435913 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:07.544462 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:07.544412 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" podStartSLOduration=1.544398816 podStartE2EDuration="1.544398816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:07.543345817 +0000 UTC m=+6031.149890314" watchObservedRunningTime="2026-11-02 21:29:07.544398816 +0000 UTC m=+6031.150943314" Nov 02 21:29:08.549351 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:08.549308 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5093a7-52ec-4a9c-be73-8b404c4ea48f" path="/var/lib/kubelet/pods/5a5093a7-52ec-4a9c-be73-8b404c4ea48f/volumes" Nov 02 21:29:10.600754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:10.599818 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:29:10.600754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:10.599882 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:18.447566 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:29:18.447489 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:29:21.003330 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:29:21.003296 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:29:51.003561 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:29:51.003523 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:30:21.003417 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:30:21.003381 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:30:35.863461 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:35.863425 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:30:40.648754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:40.648033 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:30:40.648754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:40.648160 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:40.869645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:40.867998 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" podUID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" containerName="payload-processing" containerID="cri-o://87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16" gracePeriod=25 Nov 02 21:30:41.225762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.225722 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:30:41.308318 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308284 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\") pod \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " Nov 02 21:30:41.308318 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308327 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\") pod \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " Nov 02 21:30:41.308589 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308352 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\") pod \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " Nov 02 21:30:41.308589 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308395 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\") pod \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\" (UID: \"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e\") " Nov 02 21:30:41.308808 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308780 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume" pod "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" (UID: "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:41.308897 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.308785 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume" pod "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" (UID: "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:41.310548 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.310523 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials" pod "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" (UID: "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:41.310705 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.310660 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748" pod "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" (UID: "3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e"). InnerVolumeSpecName "kube-api-access-pb748". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:41.409929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.409896 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb748\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-kube-api-access-pb748\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:41.409929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.409925 5136 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-provider-credentials\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:41.409929 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.409936 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:41.410161 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.409946 5136 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e-routing-overlay-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:41.931054 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.931024 5136 generic.go:356] "Generic (PLEG): container finished" podID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" containerID="87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16" exitCode=0 Nov 02 21:30:41.931542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.931075 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" Nov 02 21:30:41.931542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.931094 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" event={"ID":"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e","Type":"ContainerDied","Data":"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16"} Nov 02 21:30:41.931542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.931160 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-srjs9" event={"ID":"3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e","Type":"ContainerDied","Data":"d1f6c5ebdeb792d347139c9c0c66029cb86d9e55a54e0ab0420015b65ae048f7"} Nov 02 21:30:41.931542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.931189 5136 scope.go:122] "RemoveContainer" containerID="87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16" Nov 02 21:30:41.940415 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.940398 5136 scope.go:122] "RemoveContainer" containerID="87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16" Nov 02 21:30:41.940671 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:30:41.940654 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16\": container with ID starting with 87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16 not found: ID does not exist" containerID="87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16" Nov 02 21:30:41.940721 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:41.940679 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16"} err="failed to get container status \"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16\": rpc error: code = NotFound desc = could not find container \"87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16\": container with ID starting with 87415c23c9832503015fcc52b1cbb7bdc5bca85a9bed09ea2316f8e31b219b16 not found: ID does not exist" Nov 02 21:30:42.084849 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:42.084808 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:30:42.132172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:42.132138 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-srjs9"] Nov 02 21:30:42.545327 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:30:42.545284 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" path="/var/lib/kubelet/pods/3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e/volumes" Nov 02 21:30:51.184810 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:30:51.171839 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:31:21.003663 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:31:21.003629 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:31:51.016324 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:31:51.016252 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:31:53.147753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:31:53.146835 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:31:53.147753 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:31:53.146909 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:31:53.218754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:31:53.214851 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:31:53.218754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:31:53.214937 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:32:21.144237 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:32:21.143987 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:32:38.148754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:32:38.147855 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:32:38.148754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:32:38.147941 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:32:38.215911 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:32:38.215861 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:32:38.216257 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:32:38.216217 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:32:51.003020 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:32:51.002974 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:33:10.643888 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:33:10.643826 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:33:10.644706 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:33:10.644658 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:21.002877 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:33:21.002843 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:33:51.003026 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:33:51.002946 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:34:21.179110 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:34:21.179064 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:34:51.004157 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:34:51.004122 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:35:20.027812 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:20.026832 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-24nqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:35:20.027812 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:20.026919 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-24nqh" podUID="fa795690-3689-45c2-b455-a1b7b58e873d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:21.002925 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:35:21.002891 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:35:50.473451 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.473404 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:35:50.474971 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.474941 5136 cpu_manager.go:414] "containerMap: removing container" podUID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" containerName="payload-processing" Nov 02 21:35:50.475183 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.475165 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" containerName="payload-processing" Nov 02 21:35:50.475402 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.475386 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba7f808-53a0-43c4-ab6d-fc87c6b3d12e" containerName="payload-processing" Nov 02 21:35:50.479149 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.479126 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.534170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.533112 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b6bfc1-openshift-default-dockercfg-ckqpw\"" Nov 02 21:35:50.669755 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.666048 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:35:50.686877 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.670252 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.687218 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.687131 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.687466 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.687446 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.687660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.687626 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zd8\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.687917 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.687898 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.688166 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.688147 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.688396 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.688375 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.688569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.688549 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.688716 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.688698 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.794047 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.793891 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.794815 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.794787 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795016 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.794995 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795160 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795143 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795332 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795315 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795463 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795447 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795598 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795581 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795722 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795702 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zd8\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.795925 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.795907 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.796499 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.796478 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.797032 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.797011 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.798398 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.798360 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.798836 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.798817 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.802256 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.802212 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.802829 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.802798 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.847196 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.847059 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.858527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.858451 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zd8\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:50.859905 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:50.859852 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\") pod \"e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:51.056863 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:35:51.056768 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:35:51.103908 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:51.103871 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:51.527221 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:51.527192 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:35:51.529845 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:35:51.529816 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745dd97f_6750_45ed_b30f_5ecfe1636d57.slice/crio-aa864be001af92569a12d9118d5ec6df07078672c788348ec6d8bc5d307b5d22 WatchSource:0}: Error finding container aa864be001af92569a12d9118d5ec6df07078672c788348ec6d8bc5d307b5d22: Status 404 returned error can't find the container with id aa864be001af92569a12d9118d5ec6df07078672c788348ec6d8bc5d307b5d22 Nov 02 21:35:51.532326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:51.532289 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:35:51.532441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:51.532363 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:35:51.532441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:51.532407 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:35:52.145640 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:52.145604 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerStarted","Data":"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b"} Nov 02 21:35:52.145878 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:52.145659 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerStarted","Data":"aa864be001af92569a12d9118d5ec6df07078672c788348ec6d8bc5d307b5d22"} Nov 02 21:35:52.351428 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:52.351378 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podStartSLOduration=2.351364353 podStartE2EDuration="2.351364353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:52.303553938 +0000 UTC m=+6435.910098461" watchObservedRunningTime="2026-11-02 21:35:52.351364353 +0000 UTC m=+6435.957908850" Nov 02 21:35:53.104259 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:53.104222 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:53.108692 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:53.108667 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:53.150248 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:53.150219 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:53.151091 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:53.151075 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:35:58.680172 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.680135 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:35:58.684106 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.684077 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.735095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.735070 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:35:58.783029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.782995 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b6bfc1\"" Nov 02 21:35:58.783029 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.783011 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a1fe3a49-dockercfg-47q78\"" Nov 02 21:35:58.788697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.788657 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l296k\" (UniqueName: \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.788903 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.788788 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.841087 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.841059 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:35:58.844680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.844660 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.889786 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.889759 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l296k\" (UniqueName: \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.889918 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.889810 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.889918 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.889867 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56lrb\" (UniqueName: \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.889918 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.889886 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.890569 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.890545 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.896657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.896637 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:35:58.949445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.949402 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l296k\" (UniqueName: \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\") pod \"payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:58.991302 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.991273 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lrb\" (UniqueName: \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.991427 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.991309 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.992023 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.992004 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:58.997047 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:58.997029 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:35:59.049769 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:59.048696 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lrb\" (UniqueName: \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\") pod \"payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:59.157506 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:59.157478 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:35:59.361982 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:59.361958 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:35:59.363657 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:35:59.363631 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8200505_3b3c_44a7_9874_33a067859332.slice/crio-daa8fd512e08691659b65faa37b6188d9afe177a8f2857e9a57fffb62355f6a8 WatchSource:0}: Error finding container daa8fd512e08691659b65faa37b6188d9afe177a8f2857e9a57fffb62355f6a8: Status 404 returned error can't find the container with id daa8fd512e08691659b65faa37b6188d9afe177a8f2857e9a57fffb62355f6a8 Nov 02 21:35:59.519108 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:35:59.519083 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:35:59.520528 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:35:59.520499 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0c6110_0f3c_4f37_81f2_69efb0cef0d8.slice/crio-37538f0bc0d06d66ec98358c72a76d5789814d8bc36bf21991897cc58feba4f2 WatchSource:0}: Error finding container 37538f0bc0d06d66ec98358c72a76d5789814d8bc36bf21991897cc58feba4f2: Status 404 returned error can't find the container with id 37538f0bc0d06d66ec98358c72a76d5789814d8bc36bf21991897cc58feba4f2 Nov 02 21:36:00.189433 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.189394 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" event={"ID":"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8","Type":"ContainerStarted","Data":"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e"} Nov 02 21:36:00.189906 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.189463 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" event={"ID":"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8","Type":"ContainerStarted","Data":"37538f0bc0d06d66ec98358c72a76d5789814d8bc36bf21991897cc58feba4f2"} Nov 02 21:36:00.189906 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.189488 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:36:00.190835 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.190800 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" event={"ID":"c8200505-3b3c-44a7-9874-33a067859332","Type":"ContainerStarted","Data":"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30"} Nov 02 21:36:00.190987 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.190852 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" event={"ID":"c8200505-3b3c-44a7-9874-33a067859332","Type":"ContainerStarted","Data":"daa8fd512e08691659b65faa37b6188d9afe177a8f2857e9a57fffb62355f6a8"} Nov 02 21:36:00.190987 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.190942 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:36:00.297334 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.297285 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" podStartSLOduration=2.297270739 podStartE2EDuration="2.297270739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:00.297113994 +0000 UTC m=+6443.903658493" watchObservedRunningTime="2026-11-02 21:36:00.297270739 +0000 UTC m=+6443.903815234" Nov 02 21:36:00.405028 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.404972 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" podStartSLOduration=2.404955135 podStartE2EDuration="2.404955135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:00.403653517 +0000 UTC m=+6444.010198015" watchObservedRunningTime="2026-11-02 21:36:00.404955135 +0000 UTC m=+6444.011499634" Nov 02 21:36:00.752197 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.752159 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:36:00.755783 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.755757 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.806878 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.806846 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b6bfc1-serving-cert\"" Nov 02 21:36:00.809763 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.809494 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:36:00.809947 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.809902 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.810072 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.810047 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpj7c\" (UniqueName: \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.810143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.810127 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.810186 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.810171 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.911013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.910977 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.911182 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.911025 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.911182 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.911089 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.911271 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.911182 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpj7c\" (UniqueName: \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.911312 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:36:00.911290 5136 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-serving-cert: secret "maas-api-e2e-worker-single-main-b6bfc1-serving-cert" not found Nov 02 21:36:00.911401 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:36:00.911369 5136 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls podName:e08bd87a-9e71-4221-9604-6fe8e6ab0e3d nodeName:}" failed. No retries permitted until 2026-11-02 21:36:01.411346662 +0000 UTC m=+6445.017891151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls") pod "maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" (UID: "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d") : secret "maas-api-e2e-worker-single-main-b6bfc1-serving-cert" not found Nov 02 21:36:00.911772 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.911753 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.913500 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.913474 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:00.968028 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:00.967999 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpj7c\" (UniqueName: \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:01.416324 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:01.416286 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:01.418761 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:01.418725 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:01.669421 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:01.669338 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:02.034455 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:02.034423 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:36:02.036603 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:36:02.036562 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08bd87a_9e71_4221_9604_6fe8e6ab0e3d.slice/crio-e0df6356f6c175b52d2f361df782b4a3c06ae54f45d3743fc066f567694b8ceb WatchSource:0}: Error finding container e0df6356f6c175b52d2f361df782b4a3c06ae54f45d3743fc066f567694b8ceb: Status 404 returned error can't find the container with id e0df6356f6c175b52d2f361df782b4a3c06ae54f45d3743fc066f567694b8ceb Nov 02 21:36:02.038284 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:02.038263 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:36:02.201993 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:02.201958 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" event={"ID":"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d","Type":"ContainerStarted","Data":"e0df6356f6c175b52d2f361df782b4a3c06ae54f45d3743fc066f567694b8ceb"} Nov 02 21:36:03.209036 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:03.208995 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" event={"ID":"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d","Type":"ContainerStarted","Data":"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7"} Nov 02 21:36:03.209522 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:03.209204 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:09.219931 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:09.219895 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:36:09.329199 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:09.329142 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" podStartSLOduration=8.398785704 podStartE2EDuration="9.329128004s" totalImagesPullingTime="930.3423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:03.318535883 +0000 UTC m=+6446.925080382" watchObservedRunningTime="2026-11-02 21:36:09.329128004 +0000 UTC m=+6452.935672502" Nov 02 21:36:11.213779 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:11.213708 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:36:11.214841 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:11.214819 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:36:18.286441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.286410 5136 generic.go:356] "Generic (PLEG): container finished" podID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerID="ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b" exitCode=0 Nov 02 21:36:18.286921 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.286486 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerDied","Data":"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b"} Nov 02 21:36:18.286921 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.286864 5136 scope.go:122] "RemoveContainer" containerID="ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b" Nov 02 21:36:18.288023 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.287993 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:36:18.288136 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.288056 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:36:18.288136 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:18.288112 5136 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 21:36:19.316601 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:19.316558 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerStarted","Data":"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404"} Nov 02 21:36:20.109771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:20.109306 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:36:20.109771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:20.109440 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 21:36:20.109771 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:20.109486 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 21:36:21.025665 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:36:21.025626 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:36:21.107176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:21.107138 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 21:36:21.107471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:21.107443 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 21:36:22.105564 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:22.105108 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 21:36:22.105564 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:22.105187 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 21:36:23.107165 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:23.107124 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 21:36:23.107821 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:23.107788 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 21:36:23.151038 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:23.151006 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:36:24.108756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.108179 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 21:36:24.108756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.108262 5136 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 21:36:24.507212 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.507177 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:36:24.510267 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.510242 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573045 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573001 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573246 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573079 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvth\" (UniqueName: \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573246 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573139 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573246 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573173 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573246 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573203 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.573518 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.573253 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.586706 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.586664 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:36:24.608874 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.608845 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b6bfc1\"/\"default-dockercfg-sgzlw\"" Nov 02 21:36:24.609048 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.608877 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b6bfc1\"/\"kube-root-ca.crt\"" Nov 02 21:36:24.609048 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.608876 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b6bfc1\"/\"openshift-service-ca.crt\"" Nov 02 21:36:24.609048 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.608944 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b6bfc1\"/\"simulator-single-main-b6bfc1-kserve-self-signed-certs\"" Nov 02 21:36:24.674643 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674606 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.674882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674657 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvth\" (UniqueName: \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.674882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674702 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.674882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674757 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.674882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674783 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.674882 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.674822 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.675320 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.675297 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.675410 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.675351 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.675410 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.675391 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.677201 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.677179 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.677321 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.677226 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.728544 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.728513 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvth\" (UniqueName: \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\") pod \"simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:24.822654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:24.822582 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:25.108502 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.108422 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:36:25.109340 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.109321 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:36:25.192200 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:36:25.192170 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff21583f_6b29_4cb3_884f_9e6baaa547ca.slice/crio-82b8b5d547f7c9b5d789d761aff6df10a4dd611f6cb4553468db994bed978788 WatchSource:0}: Error finding container 82b8b5d547f7c9b5d789d761aff6df10a4dd611f6cb4553468db994bed978788: Status 404 returned error can't find the container with id 82b8b5d547f7c9b5d789d761aff6df10a4dd611f6cb4553468db994bed978788 Nov 02 21:36:25.212704 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.212677 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:36:25.352136 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.352088 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" event={"ID":"ff21583f-6b29-4cb3-884f-9e6baaa547ca","Type":"ContainerStarted","Data":"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3"} Nov 02 21:36:25.352346 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.352152 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" event={"ID":"ff21583f-6b29-4cb3-884f-9e6baaa547ca","Type":"ContainerStarted","Data":"82b8b5d547f7c9b5d789d761aff6df10a4dd611f6cb4553468db994bed978788"} Nov 02 21:36:25.462333 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.462282 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" podStartSLOduration=1.4622673339999999 podStartE2EDuration="1.462267334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:25.461168183 +0000 UTC m=+6469.067712682" watchObservedRunningTime="2026-11-02 21:36:25.462267334 +0000 UTC m=+6469.068811832" Nov 02 21:36:25.822813 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:25.822716 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:34.823027 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:34.822989 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:34.835163 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:34.835131 5136 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:35.413314 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:36:35.413286 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:36:51.003648 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:36:51.003575 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:37:21.004204 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:21.004165 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:37:26.434721 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:26.434685 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:37:27.065613 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:27.065578 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:37:27.170542 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:27.170510 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:37:31.142533 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:31.142498 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:37:31.143006 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:31.142691 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" podUID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" containerName="maas-api" containerID="cri-o://fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7" gracePeriod=30 Nov 02 21:37:32.066610 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.066556 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" containerID="cri-o://46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e" gracePeriod=25 Nov 02 21:37:32.171781 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.171745 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" containerID="cri-o://54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30" gracePeriod=25 Nov 02 21:37:32.401121 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.401097 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:37:32.437867 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.437837 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\") pod \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " Nov 02 21:37:32.437988 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.437881 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\" (UniqueName: \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\") pod \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " Nov 02 21:37:32.437988 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.437965 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") pod \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " Nov 02 21:37:32.438079 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.438060 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\") pod \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\" (UID: \"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d\") " Nov 02 21:37:32.438418 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.438394 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca" pod "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" (UID: "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:32.440041 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.440012 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs" pod "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" (UID: "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:32.440152 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.440066 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls" pod "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" (UID: "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:32.440211 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.440146 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c" pod "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" (UID: "e08bd87a-9e71-4221-9604-6fe8e6ab0e3d"). InnerVolumeSpecName "kube-api-access-tpj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:32.539071 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.539040 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:32.539195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.539078 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:32.539195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.539095 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpj7c\" (UniqueName: \"kubernetes.io/projected/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-kube-api-access-tpj7c\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:32.539195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.539110 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:32.699084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.699055 5136 generic.go:356] "Generic (PLEG): container finished" podID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" containerID="fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7" exitCode=0 Nov 02 21:37:32.699245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.699097 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" Nov 02 21:37:32.699245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.699133 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" event={"ID":"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d","Type":"ContainerDied","Data":"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7"} Nov 02 21:37:32.699245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.699184 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57" event={"ID":"e08bd87a-9e71-4221-9604-6fe8e6ab0e3d","Type":"ContainerDied","Data":"e0df6356f6c175b52d2f361df782b4a3c06ae54f45d3743fc066f567694b8ceb"} Nov 02 21:37:32.699245 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.699208 5136 scope.go:122] "RemoveContainer" containerID="fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7" Nov 02 21:37:32.708362 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.708336 5136 scope.go:122] "RemoveContainer" containerID="fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7" Nov 02 21:37:32.708605 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:32.708585 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7\": container with ID starting with fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7 not found: ID does not exist" containerID="fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7" Nov 02 21:37:32.708650 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.708617 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7"} err="failed to get container status \"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7\": rpc error: code = NotFound desc = could not find container \"fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7\": container with ID starting with fe19179e81ea0f86bca1f647939a485e88d91e0723cadda2d2bff5e44de16bd7 not found: ID does not exist" Nov 02 21:37:32.853766 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.853726 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:37:32.857311 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:32.857280 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b6bfc1-759b69dc58-7gn57"] Nov 02 21:37:34.545011 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:34.544979 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" path="/var/lib/kubelet/pods/e08bd87a-9e71-4221-9604-6fe8e6ab0e3d/volumes" Nov 02 21:37:41.198049 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.198007 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:37:41.198590 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.198557 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:37:41.455288 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.455198 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" podUID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" containerName="main" containerID="cri-o://6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3" gracePeriod=45 Nov 02 21:37:41.707890 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.707823 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:37:41.745848 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.745816 5136 generic.go:356] "Generic (PLEG): container finished" podID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" containerID="6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3" exitCode=0 Nov 02 21:37:41.745994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.745860 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" Nov 02 21:37:41.745994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.745875 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" event={"ID":"ff21583f-6b29-4cb3-884f-9e6baaa547ca","Type":"ContainerDied","Data":"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3"} Nov 02 21:37:41.745994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.745916 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49" event={"ID":"ff21583f-6b29-4cb3-884f-9e6baaa547ca","Type":"ContainerDied","Data":"82b8b5d547f7c9b5d789d761aff6df10a4dd611f6cb4553468db994bed978788"} Nov 02 21:37:41.745994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.745939 5136 scope.go:122] "RemoveContainer" containerID="6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3" Nov 02 21:37:41.754251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.754232 5136 scope.go:122] "RemoveContainer" containerID="6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3" Nov 02 21:37:41.754496 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:41.754480 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3\": container with ID starting with 6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3 not found: ID does not exist" containerID="6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3" Nov 02 21:37:41.754545 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.754503 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3"} err="failed to get container status \"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3\": rpc error: code = NotFound desc = could not find container \"6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3\": container with ID starting with 6b91f08d8c83c6cfcad305240efc6c3bc85950a99875df9f1e390cca38f7b7c3 not found: ID does not exist" Nov 02 21:37:41.832515 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832482 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.832702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832538 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\" (UniqueName: \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.832702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832648 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.832702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832680 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.832938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832779 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:41.832938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832793 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.832938 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.832829 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\") pod \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\" (UID: \"ff21583f-6b29-4cb3-884f-9e6baaa547ca\") " Nov 02 21:37:41.833387 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.833354 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:41.833506 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.833383 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:41.833762 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.833742 5136 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tmp-dir\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:41.833933 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.833911 5136 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-home\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:41.834042 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.833936 5136 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-model-cache\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:41.835140 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.835111 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:41.835539 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.835518 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:41.835609 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.835565 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth" pod "ff21583f-6b29-4cb3-884f-9e6baaa547ca" (UID: "ff21583f-6b29-4cb3-884f-9e6baaa547ca"). InnerVolumeSpecName "kube-api-access-6zvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:41.934583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.934555 5136 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff21583f-6b29-4cb3-884f-9e6baaa547ca-dshm\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:41.934583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.934580 5136 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff21583f-6b29-4cb3-884f-9e6baaa547ca-tls-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:41.934815 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:41.934591 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvth\" (UniqueName: \"kubernetes.io/projected/ff21583f-6b29-4cb3-884f-9e6baaa547ca-kube-api-access-6zvth\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:42.159657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:42.159625 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:37:42.162749 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:42.162710 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b6bfc1/simulator-single-main-b6bfc1-kserve-78657d7fb6-lvt49"] Nov 02 21:37:42.545431 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:42.545400 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" path="/var/lib/kubelet/pods/ff21583f-6b29-4cb3-884f-9e6baaa547ca/volumes" Nov 02 21:37:51.003101 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:51.003062 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:37:51.197496 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:51.197457 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:37:51.197754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:51.197436 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:37:52.319051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.319028 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:37:52.414223 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.414200 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:37:52.440018 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.439990 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\" (UniqueName: \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\") pod \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " Nov 02 21:37:52.440147 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.440040 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\") pod \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\" (UID: \"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8\") " Nov 02 21:37:52.440403 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.440380 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume" pod "2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" (UID: "2f0c6110-0f3c-4f37-81f2-69efb0cef0d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:52.440482 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.440470 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:52.442030 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.442010 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb" pod "2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" (UID: "2f0c6110-0f3c-4f37-81f2-69efb0cef0d8"). InnerVolumeSpecName "kube-api-access-56lrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:52.541113 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.541079 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\") pod \"c8200505-3b3c-44a7-9874-33a067859332\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " Nov 02 21:37:52.541251 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.541186 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\" (UniqueName: \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\") pod \"c8200505-3b3c-44a7-9874-33a067859332\" (UID: \"c8200505-3b3c-44a7-9874-33a067859332\") " Nov 02 21:37:52.541509 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.541488 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56lrb\" (UniqueName: \"kubernetes.io/projected/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8-kube-api-access-56lrb\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:52.541680 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.541657 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume" pod "c8200505-3b3c-44a7-9874-33a067859332" (UID: "c8200505-3b3c-44a7-9874-33a067859332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:52.543544 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.543516 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k" pod "c8200505-3b3c-44a7-9874-33a067859332" (UID: "c8200505-3b3c-44a7-9874-33a067859332"). InnerVolumeSpecName "kube-api-access-l296k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:52.642476 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.642406 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l296k\" (UniqueName: \"kubernetes.io/projected/c8200505-3b3c-44a7-9874-33a067859332-kube-api-access-l296k\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:52.642476 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.642430 5136 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8200505-3b3c-44a7-9874-33a067859332-plugins-config-volume\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:52.810862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.810823 5136 generic.go:356] "Generic (PLEG): container finished" podID="c8200505-3b3c-44a7-9874-33a067859332" containerID="54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30" exitCode=0 Nov 02 21:37:52.811061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.810889 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" Nov 02 21:37:52.811061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.810910 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" event={"ID":"c8200505-3b3c-44a7-9874-33a067859332","Type":"ContainerDied","Data":"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30"} Nov 02 21:37:52.811061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.811000 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs" event={"ID":"c8200505-3b3c-44a7-9874-33a067859332","Type":"ContainerDied","Data":"daa8fd512e08691659b65faa37b6188d9afe177a8f2857e9a57fffb62355f6a8"} Nov 02 21:37:52.811061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.811029 5136 scope.go:122] "RemoveContainer" containerID="54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30" Nov 02 21:37:52.812370 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.812355 5136 generic.go:356] "Generic (PLEG): container finished" podID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerID="46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e" exitCode=0 Nov 02 21:37:52.812471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.812412 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" event={"ID":"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8","Type":"ContainerDied","Data":"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e"} Nov 02 21:37:52.812471 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.812453 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" event={"ID":"2f0c6110-0f3c-4f37-81f2-69efb0cef0d8","Type":"ContainerDied","Data":"37538f0bc0d06d66ec98358c72a76d5789814d8bc36bf21991897cc58feba4f2"} Nov 02 21:37:52.812597 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.812420 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz" Nov 02 21:37:52.820365 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.820344 5136 scope.go:122] "RemoveContainer" containerID="54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30" Nov 02 21:37:52.820650 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:52.820632 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30\": container with ID starting with 54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30 not found: ID does not exist" containerID="54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30" Nov 02 21:37:52.820701 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.820657 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30"} err="failed to get container status \"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30\": rpc error: code = NotFound desc = could not find container \"54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30\": container with ID starting with 54d39ead0cb735d4bc38208f2c2e13ad6faebd7f99bd7e6b4b5135c34f00bc30 not found: ID does not exist" Nov 02 21:37:52.820701 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.820674 5136 scope.go:122] "RemoveContainer" containerID="46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e" Nov 02 21:37:52.828272 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.828254 5136 scope.go:122] "RemoveContainer" containerID="46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e" Nov 02 21:37:52.828536 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:37:52.828519 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e\": container with ID starting with 46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e not found: ID does not exist" containerID="46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e" Nov 02 21:37:52.828577 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.828542 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e"} err="failed to get container status \"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e\": rpc error: code = NotFound desc = could not find container \"46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e\": container with ID starting with 46366ed61290fe4f175678d768954e7a4a0c639a3020eaef45cf143b55d6350e not found: ID does not exist" Nov 02 21:37:52.965434 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.965411 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:37:52.968949 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:52.968918 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b6bfc1-6bd956c5dk7hlz"] Nov 02 21:37:53.073381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:53.073351 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:37:53.076564 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:53.076536 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b6bfc1-85796ntcvs"] Nov 02 21:37:54.545551 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:54.545518 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" path="/var/lib/kubelet/pods/2f0c6110-0f3c-4f37-81f2-69efb0cef0d8/volumes" Nov 02 21:37:54.546058 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:37:54.545911 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8200505-3b3c-44a7-9874-33a067859332" path="/var/lib/kubelet/pods/c8200505-3b3c-44a7-9874-33a067859332/volumes" Nov 02 21:38:05.596437 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:05.596405 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:38:05.596856 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:05.596614 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" containerID="cri-o://18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404" gracePeriod=30 Nov 02 21:38:10.111638 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.111599 5136 patch_prober.go:29] interesting pod/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:10.112133 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.111662 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:10.855183 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.855160 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:38:10.899258 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.899188 5136 generic.go:356] "Generic (PLEG): container finished" podID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerID="18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404" exitCode=0 Nov 02 21:38:10.899258 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.899247 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" Nov 02 21:38:10.899445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.899275 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerDied","Data":"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404"} Nov 02 21:38:10.899445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.899326 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq" event={"ID":"745dd97f-6750-45ed-b30f-5ecfe1636d57","Type":"ContainerDied","Data":"aa864be001af92569a12d9118d5ec6df07078672c788348ec6d8bc5d307b5d22"} Nov 02 21:38:10.899445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.899346 5136 scope.go:122] "RemoveContainer" containerID="18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404" Nov 02 21:38:10.907960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.907940 5136 scope.go:122] "RemoveContainer" containerID="ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b" Nov 02 21:38:10.912150 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912131 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912185 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912214 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912245 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912282 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912271 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912337 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912374 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912443 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912507 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\") pod \"745dd97f-6750-45ed-b30f-5ecfe1636d57\" (UID: \"745dd97f-6750-45ed-b30f-5ecfe1636d57\") " Nov 02 21:38:10.912765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912525 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.912765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912532 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.912765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912745 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.912953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912772 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:10.912953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912896 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.912953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912923 5136 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.912953 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912946 5136 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-credential-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.913195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912962 5136 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-workload-socket\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.913195 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.912974 5136 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745dd97f-6750-45ed-b30f-5ecfe1636d57-istiod-ca-cert\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.914624 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.914598 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "kube-api-access-n2zd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:10.914767 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.914718 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:10.914841 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.914788 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.914884 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.914855 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo" pod "745dd97f-6750-45ed-b30f-5ecfe1636d57" (UID: "745dd97f-6750-45ed-b30f-5ecfe1636d57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:10.916015 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.915999 5136 scope.go:122] "RemoveContainer" containerID="18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404" Nov 02 21:38:10.916304 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:38:10.916282 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404\": container with ID starting with 18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404 not found: ID does not exist" containerID="18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404" Nov 02 21:38:10.916360 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.916312 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404"} err="failed to get container status \"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404\": rpc error: code = NotFound desc = could not find container \"18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404\": container with ID starting with 18e220b15b624657dada46a3247be464aaf81ac71018e234f6a28cdb636ac404 not found: ID does not exist" Nov 02 21:38:10.916360 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.916330 5136 scope.go:122] "RemoveContainer" containerID="ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b" Nov 02 21:38:10.916588 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:38:10.916570 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b\": container with ID starting with ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b not found: ID does not exist" containerID="ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b" Nov 02 21:38:10.916632 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:10.916594 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b"} err="failed to get container status \"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b\": rpc error: code = NotFound desc = could not find container \"ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b\": container with ID starting with ea572951ecce141ab17270b5d94b8d66ddc887a35c49a8a5bba8fb3a81a6fd7b not found: ID does not exist" Nov 02 21:38:11.014510 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.014469 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zd8\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-kube-api-access-n2zd8\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.014667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.014515 5136 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-podinfo\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.014667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.014532 5136 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-envoy\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.014667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.014544 5136 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-token\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.014667 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.014556 5136 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745dd97f-6750-45ed-b30f-5ecfe1636d57-istio-data\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.313068 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.313033 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:38:11.320665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:11.320490 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b6bfc1-openshift-default-7f49469bd5pvnjq"] Nov 02 21:38:12.545721 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:38:12.545690 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" path="/var/lib/kubelet/pods/745dd97f-6750-45ed-b30f-5ecfe1636d57/volumes" Nov 02 21:38:21.003295 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:38:21.003218 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:38:51.003966 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:38:51.003931 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:39:21.004604 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:39:21.004549 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:39:51.002487 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:39:51.002407 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:40:10.642754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:10.639860 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:40:10.642754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:10.639935 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-ns7hq" podUID="f17725dd-2440-4da1-93ec-41e643aaf992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:40:21.002502 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:40:21.002469 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:40:29.659253 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659219 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-mfgb7"] Nov 02 21:40:29.659660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659636 5136 cpu_manager.go:414] "containerMap: removing container" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.659660 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659650 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659670 5136 cpu_manager.go:414] "containerMap: removing container" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659676 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659693 5136 cpu_manager.go:414] "containerMap: removing container" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659698 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659709 5136 cpu_manager.go:414] "containerMap: removing container" podUID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" containerName="main" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659713 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" containerName="main" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659722 5136 cpu_manager.go:414] "containerMap: removing container" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" Nov 02 21:40:29.659751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659750 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659758 5136 cpu_manager.go:414] "containerMap: removing container" podUID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" containerName="maas-api" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659763 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" containerName="maas-api" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659821 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659830 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="745dd97f-6750-45ed-b30f-5ecfe1636d57" containerName="istio-proxy" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659838 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8200505-3b3c-44a7-9874-33a067859332" containerName="payload-pre-processing" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659845 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff21583f-6b29-4cb3-884f-9e6baaa547ca" containerName="main" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659853 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0c6110-0f3c-4f37-81f2-69efb0cef0d8" containerName="payload-processing" Nov 02 21:40:29.660009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.659859 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08bd87a-9e71-4221-9604-6fe8e6ab0e3d" containerName="maas-api" Nov 02 21:40:29.663446 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.663415 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.762153 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.762121 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-mfgb7"] Nov 02 21:40:29.791605 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.791581 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-metrics-certs\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.791805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.791614 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf79ad65-8ce1-4368-915e-2cdaf1390962-openshift-service-ca\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.791805 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.791654 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-maas-api-tls\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.791950 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.791867 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjvh\" (UniqueName: \"kubernetes.io/projected/bf79ad65-8ce1-4368-915e-2cdaf1390962-kube-api-access-5qjvh\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.893222 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.893191 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjvh\" (UniqueName: \"kubernetes.io/projected/bf79ad65-8ce1-4368-915e-2cdaf1390962-kube-api-access-5qjvh\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.893381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.893255 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-metrics-certs\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.893381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.893278 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf79ad65-8ce1-4368-915e-2cdaf1390962-openshift-service-ca\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.893381 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.893336 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-maas-api-tls\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.894163 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.894136 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf79ad65-8ce1-4368-915e-2cdaf1390962-openshift-service-ca\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.896393 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.896364 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-maas-api-tls\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.896518 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.896406 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf79ad65-8ce1-4368-915e-2cdaf1390962-metrics-certs\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.947528 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.947496 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjvh\" (UniqueName: \"kubernetes.io/projected/bf79ad65-8ce1-4368-915e-2cdaf1390962-kube-api-access-5qjvh\") pod \"maas-api-54b974b655-mfgb7\" (UID: \"bf79ad65-8ce1-4368-915e-2cdaf1390962\") " pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:29.976721 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:29.976695 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:30.364526 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:30.364503 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b974b655-mfgb7"] Nov 02 21:40:30.366371 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:40:30.366344 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf79ad65_8ce1_4368_915e_2cdaf1390962.slice/crio-740ff7384b327fa208453c9311681581fb7cff520e1c3535430be19228d91e66 WatchSource:0}: Error finding container 740ff7384b327fa208453c9311681581fb7cff520e1c3535430be19228d91e66: Status 404 returned error can't find the container with id 740ff7384b327fa208453c9311681581fb7cff520e1c3535430be19228d91e66 Nov 02 21:40:30.594051 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:30.594012 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" event={"ID":"bf79ad65-8ce1-4368-915e-2cdaf1390962","Type":"ContainerStarted","Data":"740ff7384b327fa208453c9311681581fb7cff520e1c3535430be19228d91e66"} Nov 02 21:40:31.607360 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:31.607254 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" event={"ID":"bf79ad65-8ce1-4368-915e-2cdaf1390962","Type":"ContainerStarted","Data":"b69ae2f1387e9ff0fa3c4626413453a468e7aa0b0f5ab18b8ed3faab98e12a6a"} Nov 02 21:40:31.607858 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:31.607369 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:31.717584 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:31.717532 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" podStartSLOduration=1.729849465 podStartE2EDuration="2.717519683s" totalImagesPullingTime="987.670218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:31.717291583 +0000 UTC m=+6715.323836083" watchObservedRunningTime="2026-11-02 21:40:31.717519683 +0000 UTC m=+6715.324064180" Nov 02 21:40:37.616647 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:37.616617 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b974b655-mfgb7" Nov 02 21:40:45.945898 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:45.945867 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 21:40:45.946534 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:45.946109 5136 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" podUID="188c464b-11a9-452f-8b45-de935f8b28fb" containerName="maas-api" containerID="cri-o://9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f" gracePeriod=30 Nov 02 21:40:47.199802 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.199775 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 21:40:47.281537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281466 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\") pod \"188c464b-11a9-452f-8b45-de935f8b28fb\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " Nov 02 21:40:47.281537 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281518 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\") pod \"188c464b-11a9-452f-8b45-de935f8b28fb\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " Nov 02 21:40:47.281808 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281554 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\") pod \"188c464b-11a9-452f-8b45-de935f8b28fb\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " Nov 02 21:40:47.281808 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281644 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\" (UniqueName: \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\") pod \"188c464b-11a9-452f-8b45-de935f8b28fb\" (UID: \"188c464b-11a9-452f-8b45-de935f8b28fb\") " Nov 02 21:40:47.281943 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281856 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca" pod "188c464b-11a9-452f-8b45-de935f8b28fb" (UID: "188c464b-11a9-452f-8b45-de935f8b28fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:47.281998 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.281943 5136 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c464b-11a9-452f-8b45-de935f8b28fb-openshift-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:47.283748 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.283709 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls" pod "188c464b-11a9-452f-8b45-de935f8b28fb" (UID: "188c464b-11a9-452f-8b45-de935f8b28fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:47.283748 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.283721 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs" pod "188c464b-11a9-452f-8b45-de935f8b28fb" (UID: "188c464b-11a9-452f-8b45-de935f8b28fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:47.283910 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.283879 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m" pod "188c464b-11a9-452f-8b45-de935f8b28fb" (UID: "188c464b-11a9-452f-8b45-de935f8b28fb"). InnerVolumeSpecName "kube-api-access-rkl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:47.383326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.383293 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkl8m\" (UniqueName: \"kubernetes.io/projected/188c464b-11a9-452f-8b45-de935f8b28fb-kube-api-access-rkl8m\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:47.383326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.383323 5136 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-metrics-certs\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:47.383326 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.383333 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188c464b-11a9-452f-8b45-de935f8b28fb-maas-api-tls\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:47.708089 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.708055 5136 generic.go:356] "Generic (PLEG): container finished" podID="188c464b-11a9-452f-8b45-de935f8b28fb" containerID="9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f" exitCode=0 Nov 02 21:40:47.708300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.708110 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" Nov 02 21:40:47.708300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.708145 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" event={"ID":"188c464b-11a9-452f-8b45-de935f8b28fb","Type":"ContainerDied","Data":"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f"} Nov 02 21:40:47.708300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.708207 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj" event={"ID":"188c464b-11a9-452f-8b45-de935f8b28fb","Type":"ContainerDied","Data":"bfaaa626db14e97bc5fa4c85011c3cad657262ea77351aa70cf9d436b4779b68"} Nov 02 21:40:47.708300 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.708236 5136 scope.go:122] "RemoveContainer" containerID="9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f" Nov 02 21:40:47.717719 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.717704 5136 scope.go:122] "RemoveContainer" containerID="9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f" Nov 02 21:40:47.718088 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:40:47.718071 5136 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f\": container with ID starting with 9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f not found: ID does not exist" containerID="9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f" Nov 02 21:40:47.718142 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.718096 5136 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f"} err="failed to get container status \"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f\": rpc error: code = NotFound desc = could not find container \"9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f\": container with ID starting with 9bde70d01e46a9d7bf578598c74a70c760801149d1deb972c8459b501c87eb2f not found: ID does not exist" Nov 02 21:40:47.862290 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.862257 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 21:40:47.866238 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:47.866214 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-794d57d6f9-5v8sj"] Nov 02 21:40:48.545452 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:40:48.545418 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188c464b-11a9-452f-8b45-de935f8b28fb" path="/var/lib/kubelet/pods/188c464b-11a9-452f-8b45-de935f8b28fb/volumes" Nov 02 21:40:51.173538 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:40:51.173492 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:41:21.003320 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:41:21.003273 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:41:51.008245 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:41:51.008198 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:42:21.002997 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:42:21.002959 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:42:51.003287 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:42:51.003209 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:43:21.003332 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:43:21.003292 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:43:51.003105 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:43:51.003070 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:44:21.002932 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:44:21.002840 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:44:51.003845 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:44:51.003812 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:45:00.151131 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.151098 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:00.151654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.151558 5136 cpu_manager.go:414] "containerMap: removing container" podUID="188c464b-11a9-452f-8b45-de935f8b28fb" containerName="maas-api" Nov 02 21:45:00.151654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.151569 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188c464b-11a9-452f-8b45-de935f8b28fb" containerName="maas-api" Nov 02 21:45:00.151654 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.151635 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="188c464b-11a9-452f-8b45-de935f8b28fb" containerName="maas-api" Nov 02 21:45:00.153994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.153973 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.252035 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.252000 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:00.253884 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.253861 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:00.271441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.271410 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.271441 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.271449 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7hd\" (UniqueName: \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.372924 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.372893 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.372924 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.372928 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7hd\" (UniqueName: \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.373515 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.373497 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.426961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.426896 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7hd\" (UniqueName: \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\") pod \"maas-api-key-cleanup-29894265-ppths\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.465790 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.465762 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:00.826213 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:00.826190 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:00.828552 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:45:00.828514 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30ab8fd_b7f8_4d89_a34e_c462f9782afa.slice/crio-3afd6852093d4c859fbd4d63997ddd4803c041466a8ca9a49d84ff7451e7bfab WatchSource:0}: Error finding container 3afd6852093d4c859fbd4d63997ddd4803c041466a8ca9a49d84ff7451e7bfab: Status 404 returned error can't find the container with id 3afd6852093d4c859fbd4d63997ddd4803c041466a8ca9a49d84ff7451e7bfab Nov 02 21:45:01.319974 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:01.319943 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerStarted","Data":"6f6df009280a38efba5bebaff2b40c89a27f3b7dfcc0055b610f742b17a37ed3"} Nov 02 21:45:01.320382 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:01.320001 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerStarted","Data":"3afd6852093d4c859fbd4d63997ddd4803c041466a8ca9a49d84ff7451e7bfab"} Nov 02 21:45:01.427087 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:01.426976 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" podStartSLOduration=1.426961821 podStartE2EDuration="1.426961821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:01.426924782 +0000 UTC m=+6985.033469277" watchObservedRunningTime="2026-11-02 21:45:01.426961821 +0000 UTC m=+6985.033506665" Nov 02 21:45:02.329163 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:02.329112 5136 generic.go:356] "Generic (PLEG): container finished" podID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerID="6f6df009280a38efba5bebaff2b40c89a27f3b7dfcc0055b610f742b17a37ed3" exitCode=60 Nov 02 21:45:02.329703 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:02.329187 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerDied","Data":"6f6df009280a38efba5bebaff2b40c89a27f3b7dfcc0055b610f742b17a37ed3"} Nov 02 21:45:02.329703 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:02.329511 5136 scope.go:122] "RemoveContainer" containerID="6f6df009280a38efba5bebaff2b40c89a27f3b7dfcc0055b610f742b17a37ed3" Nov 02 21:45:03.438672 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:03.420411 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerStarted","Data":"9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5"} Nov 02 21:45:04.430652 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:04.430618 5136 generic.go:356] "Generic (PLEG): container finished" podID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerID="9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5" exitCode=60 Nov 02 21:45:04.431114 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:04.431087 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerDied","Data":"9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5"} Nov 02 21:45:04.431336 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:04.431305 5136 scope.go:122] "RemoveContainer" containerID="6f6df009280a38efba5bebaff2b40c89a27f3b7dfcc0055b610f742b17a37ed3" Nov 02 21:45:04.432074 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:04.432057 5136 scope.go:122] "RemoveContainer" containerID="9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5" Nov 02 21:45:04.432635 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:45:04.432614 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ppths_odh-ai-gateway-infra(b30ab8fd-b7f8-4d89-a34e-c462f9782afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" Nov 02 21:45:05.438697 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:05.438668 5136 scope.go:122] "RemoveContainer" containerID="9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5" Nov 02 21:45:05.439108 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:45:05.438896 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ppths_odh-ai-gateway-infra(b30ab8fd-b7f8-4d89-a34e-c462f9782afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" Nov 02 21:45:15.542754 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:15.542200 5136 scope.go:122] "RemoveContainer" containerID="9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5" Nov 02 21:45:16.510126 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:16.509201 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerStarted","Data":"05fb6153e1ce9aa8cceb361311d57374ebf2c295a819eed2ecaf6d8a8880e89e"} Nov 02 21:45:17.516077 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:17.516044 5136 generic.go:356] "Generic (PLEG): container finished" podID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerID="05fb6153e1ce9aa8cceb361311d57374ebf2c295a819eed2ecaf6d8a8880e89e" exitCode=60 Nov 02 21:45:17.516521 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:17.516122 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerDied","Data":"05fb6153e1ce9aa8cceb361311d57374ebf2c295a819eed2ecaf6d8a8880e89e"} Nov 02 21:45:17.516521 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:17.516184 5136 scope.go:122] "RemoveContainer" containerID="9018aefbd8c529509455bd822314972c0d74b7b52337e8f28056e1d081ea06c5" Nov 02 21:45:17.516521 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:17.516480 5136 scope.go:122] "RemoveContainer" containerID="05fb6153e1ce9aa8cceb361311d57374ebf2c295a819eed2ecaf6d8a8880e89e" Nov 02 21:45:17.516745 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:45:17.516713 5136 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ppths_odh-ai-gateway-infra(b30ab8fd-b7f8-4d89-a34e-c462f9782afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" Nov 02 21:45:17.628185 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:17.628157 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:18.656314 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.656292 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:18.724845 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.724822 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\" (UniqueName: \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\") pod \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " Nov 02 21:45:18.724992 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.724866 5136 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\") pod \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\" (UID: \"b30ab8fd-b7f8-4d89-a34e-c462f9782afa\") " Nov 02 21:45:18.725247 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.725225 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca" pod "b30ab8fd-b7f8-4d89-a34e-c462f9782afa" (UID: "b30ab8fd-b7f8-4d89-a34e-c462f9782afa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:18.726954 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.726930 5136 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd" pod "b30ab8fd-b7f8-4d89-a34e-c462f9782afa" (UID: "b30ab8fd-b7f8-4d89-a34e-c462f9782afa"). InnerVolumeSpecName "kube-api-access-xw7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:18.825960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.825880 5136 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7hd\" (UniqueName: \"kubernetes.io/projected/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-kube-api-access-xw7hd\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:18.825960 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:18.825915 5136 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30ab8fd-b7f8-4d89-a34e-c462f9782afa-maas-api-service-ca\") on node \"ip-10-0-128-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.528350 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:19.528321 5136 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" Nov 02 21:45:19.528540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:19.528358 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths" event={"ID":"b30ab8fd-b7f8-4d89-a34e-c462f9782afa","Type":"ContainerDied","Data":"3afd6852093d4c859fbd4d63997ddd4803c041466a8ca9a49d84ff7451e7bfab"} Nov 02 21:45:19.528540 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:19.528437 5136 scope.go:122] "RemoveContainer" containerID="05fb6153e1ce9aa8cceb361311d57374ebf2c295a819eed2ecaf6d8a8880e89e" Nov 02 21:45:19.682791 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:19.682758 5136 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:19.730105 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:19.730076 5136 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ppths"] Nov 02 21:45:20.545958 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:45:20.545924 5136 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" path="/var/lib/kubelet/pods/b30ab8fd-b7f8-4d89-a34e-c462f9782afa/volumes" Nov 02 21:45:21.003637 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:45:21.003598 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:45:51.004138 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:45:51.004042 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:46:08.153902 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:08.153856 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nzffk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:46:08.154615 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:08.154580 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nzffk" podUID="3d7346bd-7650-4820-a6e3-65ef8979ff89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:46:08.215902 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:08.215852 5136 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z44pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:46:08.216219 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:08.216184 5136 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z44pn" podUID="14f88a84-e518-4c2c-abbd-8bc07a283a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:46:21.003030 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:46:21.002994 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:46:51.003973 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:46:51.003939 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:46:55.873095 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:55.873068 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656cc65bfd-xlcwc_78556d8e-59a8-4977-8d09-8586ee3cb6c7/manager/0.log" Nov 02 21:46:56.500126 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:56.500096 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mg6sc_95efd56e-e640-4249-b8f4-5e55d11f91c2/manager/0.log" Nov 02 21:46:57.259800 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:57.259767 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-5f2xx_c157ba0d-6496-4ec6-bfad-aafbaf9439f3/manager/0.log" Nov 02 21:46:57.600493 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:57.600412 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w9xl7_73e480fc-41e8-405f-a466-2e08f9041414/server/0.log" Nov 02 21:46:57.821371 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:57.821342 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2slch_186a3e83-bf4f-4cca-972d-2761964a7c63/manager/0.log" Nov 02 21:46:58.072056 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:58.072017 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-jb6hq_78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b/proxy/0.log" Nov 02 21:46:59.033266 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:46:59.033238 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b974b655-mfgb7_bf79ad65-8ce1-4368-915e-2cdaf1390962/maas-api/0.log" Nov 02 21:47:03.138049 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:03.138005 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-wttbq_1a9fc457-ac3f-4440-b120-10b514edc6a5/postgres/0.log" Nov 02 21:47:05.757201 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:05.757170 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7cwmg_ba3a6e88-66b2-4b55-a673-6260a7cbbe1a/manager/0.log" Nov 02 21:47:06.028756 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:06.028670 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pppkd_0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff/kuadrant-console-plugin/0.log" Nov 02 21:47:08.987510 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:08.987477 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9c5d7d8f-bs77n_46cd203e-7ee4-4aa9-8b62-c5de1807bb2c/kube-auth-proxy/0.log" Nov 02 21:47:09.430785 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:09.430689 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-24nqh_fa795690-3689-45c2-b455-a1b7b58e873d/istio-proxy/0.log" Nov 02 21:47:11.266173 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:11.266145 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-ns7hq_f17725dd-2440-4da1-93ec-41e643aaf992/istio-proxy/0.log" Nov 02 21:47:11.577084 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:11.577010 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-nzffk_3d7346bd-7650-4820-a6e3-65ef8979ff89/istio-proxy/0.log" Nov 02 21:47:12.164942 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:12.164904 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-z44pn_14f88a84-e518-4c2c-abbd-8bc07a283a4d/istio-proxy/0.log" Nov 02 21:47:12.714357 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:12.714322 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668545c68-6zrh6_04992037-54d2-409a-9364-9d1acab24b15/payload-pre-processing/0.log" Nov 02 21:47:12.934539 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:12.934504 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7f8f4d5bdc-f56nj_6c777b53-481c-40bf-a1c2-d2c188988730/payload-processing/0.log" Nov 02 21:47:14.305093 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:14.305067 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-x7tgc_604a1fb3-761d-442c-8ef7-076a37af476d/main/0.log" Nov 02 21:47:15.105054 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:15.105023 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-z2zkj_46d6c4bb-3db2-4846-9a60-ada0cd07e351/main/0.log" Nov 02 21:47:15.642961 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:15.642933 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4lsbj_8ec662da-351c-4ed1-a35b-1a0fea0f294c/main/0.log" Nov 02 21:47:21.004003 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:47:21.003924 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:47:25.834565 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.834530 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9pw5/must-gather-4mghx"] Nov 02 21:47:25.835162 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835141 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835259 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835164 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835259 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835224 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835259 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835236 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835328 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835442 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835344 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835563 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835478 5136 cpu_manager.go:414] "containerMap: removing container" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835563 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835489 5136 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.835677 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.835593 5136 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ab8fd-b7f8-4d89-a34e-c462f9782afa" containerName="cleanup" Nov 02 21:47:25.838862 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.838834 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:25.892128 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.892090 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"openshift-service-ca.crt\"" Nov 02 21:47:25.924379 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.924348 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ftr\" (UniqueName: \"kubernetes.io/projected/e77aab03-934c-4626-bce0-4b95d5aa95a8-kube-api-access-26ftr\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:25.924544 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.924428 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e77aab03-934c-4626-bce0-4b95d5aa95a8-must-gather-output\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:25.939580 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.939560 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"kube-root-ca.crt\"" Nov 02 21:47:25.939714 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.939660 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k9pw5\"/\"default-dockercfg-sn7x5\"" Nov 02 21:47:25.941229 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.941214 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/must-gather-4mghx"] Nov 02 21:47:25.942479 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.942459 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-84fsd/must-gather-tjbv5"] Nov 02 21:47:25.945973 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.945960 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:25.991081 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.991058 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84fsd/must-gather-tjbv5"] Nov 02 21:47:25.999583 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.999544 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84fsd\"/\"kube-root-ca.crt\"" Nov 02 21:47:25.999776 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:25.999757 5136 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-84fsd\"/\"default-dockercfg-lbc6d\"" Nov 02 21:47:26.025552 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.025529 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ftr\" (UniqueName: \"kubernetes.io/projected/e77aab03-934c-4626-bce0-4b95d5aa95a8-kube-api-access-26ftr\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:26.025657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.025584 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e77aab03-934c-4626-bce0-4b95d5aa95a8-must-gather-output\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:26.025994 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.025977 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e77aab03-934c-4626-bce0-4b95d5aa95a8-must-gather-output\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:26.080674 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.080644 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ftr\" (UniqueName: \"kubernetes.io/projected/e77aab03-934c-4626-bce0-4b95d5aa95a8-kube-api-access-26ftr\") pod \"must-gather-4mghx\" (UID: \"e77aab03-934c-4626-bce0-4b95d5aa95a8\") " pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:26.098361 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.098307 5136 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84fsd\"/\"openshift-service-ca.crt\"" Nov 02 21:47:26.126289 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.126259 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-must-gather-output\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.126417 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.126296 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7csx\" (UniqueName: \"kubernetes.io/projected/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-kube-api-access-w7csx\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.153546 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.153527 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/must-gather-4mghx" Nov 02 21:47:26.227304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.227274 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-must-gather-output\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.227458 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.227320 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7csx\" (UniqueName: \"kubernetes.io/projected/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-kube-api-access-w7csx\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.227640 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.227620 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-must-gather-output\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.281481 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.281452 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7csx\" (UniqueName: \"kubernetes.io/projected/d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6-kube-api-access-w7csx\") pod \"must-gather-tjbv5\" (UID: \"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6\") " pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.527449 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.521043 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/must-gather-4mghx"] Nov 02 21:47:26.529314 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:47:26.529282 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode77aab03_934c_4626_bce0_4b95d5aa95a8.slice/crio-6375bced0911e4519c537cda8492bc413f8dfa67c17eb2f7f0c2d003b6ebbb2c WatchSource:0}: Error finding container 6375bced0911e4519c537cda8492bc413f8dfa67c17eb2f7f0c2d003b6ebbb2c: Status 404 returned error can't find the container with id 6375bced0911e4519c537cda8492bc413f8dfa67c17eb2f7f0c2d003b6ebbb2c Nov 02 21:47:26.531212 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.531190 5136 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:47:26.559627 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.559601 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84fsd/must-gather-tjbv5" Nov 02 21:47:26.922469 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:26.922445 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84fsd/must-gather-tjbv5"] Nov 02 21:47:26.924628 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:47:26.924598 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3fc41b8_84ad_4bc4_9c0d_f66ebc2130d6.slice/crio-4c26c0b3fc37eef8cc011eda9068aee61c0228f133378138fda9a60780476f85 WatchSource:0}: Error finding container 4c26c0b3fc37eef8cc011eda9068aee61c0228f133378138fda9a60780476f85: Status 404 returned error can't find the container with id 4c26c0b3fc37eef8cc011eda9068aee61c0228f133378138fda9a60780476f85 Nov 02 21:47:27.354943 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:27.354908 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/must-gather-4mghx" event={"ID":"e77aab03-934c-4626-bce0-4b95d5aa95a8","Type":"ContainerStarted","Data":"6375bced0911e4519c537cda8492bc413f8dfa67c17eb2f7f0c2d003b6ebbb2c"} Nov 02 21:47:27.356009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:27.355985 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84fsd/must-gather-tjbv5" event={"ID":"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6","Type":"ContainerStarted","Data":"4c26c0b3fc37eef8cc011eda9068aee61c0228f133378138fda9a60780476f85"} Nov 02 21:47:28.363009 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:28.362975 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/must-gather-4mghx" event={"ID":"e77aab03-934c-4626-bce0-4b95d5aa95a8","Type":"ContainerStarted","Data":"82dd3e78aae437d24bf58023169cee0a54b8ba2f42af7c23893dddd4697fce7c"} Nov 02 21:47:28.363413 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:28.363017 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/must-gather-4mghx" event={"ID":"e77aab03-934c-4626-bce0-4b95d5aa95a8","Type":"ContainerStarted","Data":"4df365d3d3887deb9c6c5fcc5bc39245c9a6472fd6f81fc2fef43e61002a239d"} Nov 02 21:47:28.470839 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:28.470788 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k9pw5/must-gather-4mghx" podStartSLOduration=2.015699084 podStartE2EDuration="3.470771459s" totalImagesPullingTime="1.455072375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:28.470032027 +0000 UTC m=+7132.076576525" watchObservedRunningTime="2026-11-02 21:47:28.470771459 +0000 UTC m=+7132.077315957" Nov 02 21:47:34.399866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:34.399783 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84fsd/must-gather-tjbv5" event={"ID":"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6","Type":"ContainerStarted","Data":"fc2c9fb889ccd491eefea59f5d170e5be7f6f86bc72b51eb20b33dd8a28dcd72"} Nov 02 21:47:34.399866 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:34.399828 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84fsd/must-gather-tjbv5" event={"ID":"d3fc41b8-84ad-4bc4-9c0d-f66ebc2130d6","Type":"ContainerStarted","Data":"6ed472ebc7c23d35e03f29ceb5861461d8f3370ee88690445482ccb8a852aded"} Nov 02 21:47:34.509690 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:34.509503 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-84fsd/must-gather-tjbv5" podStartSLOduration=2.395960856 podStartE2EDuration="9.509487479s" totalImagesPullingTime="7.113526623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:34.508038321 +0000 UTC m=+7138.114582819" watchObservedRunningTime="2026-11-02 21:47:34.509487479 +0000 UTC m=+7138.116031955" Nov 02 21:47:35.429751 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:35.429683 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h6fsm_e904a0d4-e8e9-4279-bba7-212e6b10fb5d/global-pull-secret-syncer/0.log" Nov 02 21:47:39.482364 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:39.481047 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656cc65bfd-xlcwc_78556d8e-59a8-4977-8d09-8586ee3cb6c7/manager/0.log" Nov 02 21:47:39.838590 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:39.834288 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wfdgc_b6842654-0249-4e3b-ade8-d0e46997bed4/konnectivity-agent/0.log" Nov 02 21:47:40.068936 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:40.068905 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mg6sc_95efd56e-e640-4249-b8f4-5e55d11f91c2/manager/0.log" Nov 02 21:47:40.192156 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:40.192029 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-34.us-west-2.compute.internal_d53a41f215f25804a365e963f8c63305/haproxy/0.log" Nov 02 21:47:41.147775 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:41.147744 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-5f2xx_c157ba0d-6496-4ec6-bfad-aafbaf9439f3/manager/0.log" Nov 02 21:47:41.273946 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:41.273908 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w9xl7_73e480fc-41e8-405f-a466-2e08f9041414/server/0.log" Nov 02 21:47:44.313765 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:44.313640 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2slch_186a3e83-bf4f-4cca-972d-2761964a7c63/manager/0.log" Nov 02 21:47:44.458118 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:44.458084 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-jb6hq_78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b/proxy/0.log" Nov 02 21:47:47.140176 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.140141 5136 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh"] Nov 02 21:47:47.146702 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.146673 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.197144 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.197101 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh"] Nov 02 21:47:47.208170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.208136 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-lib-modules\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.208503 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.208480 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfcf\" (UniqueName: \"kubernetes.io/projected/eee1d135-b802-45cb-a2cc-6cdcab121f18-kube-api-access-9rfcf\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.208759 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.208724 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-podres\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.208912 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.208894 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-sys\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.209052 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.209026 5136 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-proc\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310439 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310392 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-lib-modules\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310439 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310450 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfcf\" (UniqueName: \"kubernetes.io/projected/eee1d135-b802-45cb-a2cc-6cdcab121f18-kube-api-access-9rfcf\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310495 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-podres\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310525 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-sys\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310534 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-lib-modules\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310607 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-sys\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310664 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310659 5136 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-proc\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310868 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310608 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-podres\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.310868 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.310779 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eee1d135-b802-45cb-a2cc-6cdcab121f18-proc\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.364687 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.364637 5136 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfcf\" (UniqueName: \"kubernetes.io/projected/eee1d135-b802-45cb-a2cc-6cdcab121f18-kube-api-access-9rfcf\") pod \"perf-node-gather-daemonset-ztpnh\" (UID: \"eee1d135-b802-45cb-a2cc-6cdcab121f18\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.429369 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.429294 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7cwmg_ba3a6e88-66b2-4b55-a673-6260a7cbbe1a/manager/0.log" Nov 02 21:47:47.460061 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.460029 5136 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:47.824047 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.824020 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656cc65bfd-xlcwc_78556d8e-59a8-4977-8d09-8586ee3cb6c7/manager/0.log" Nov 02 21:47:47.835170 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.835142 5136 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh"] Nov 02 21:47:47.837412 ip-10-0-128-34 kubenswrapper[5136]: W1102 21:47:47.837379 5136 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeee1d135_b802_45cb_a2cc_6cdcab121f18.slice/crio-9442bbbd9451c3562b8eab21f0c42b3392670cd8ce3dc50df97ec3ae63bbbfee WatchSource:0}: Error finding container 9442bbbd9451c3562b8eab21f0c42b3392670cd8ce3dc50df97ec3ae63bbbfee: Status 404 returned error can't find the container with id 9442bbbd9451c3562b8eab21f0c42b3392670cd8ce3dc50df97ec3ae63bbbfee Nov 02 21:47:47.843249 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:47.843228 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pppkd_0c5f0991-ec7e-4cef-ac8b-43495b3ba3ff/kuadrant-console-plugin/0.log" Nov 02 21:47:48.334143 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:48.334109 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mg6sc_95efd56e-e640-4249-b8f4-5e55d11f91c2/manager/0.log" Nov 02 21:47:48.549523 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:48.549486 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" event={"ID":"eee1d135-b802-45cb-a2cc-6cdcab121f18","Type":"ContainerStarted","Data":"66f3167b4f1464a4684cb30988c72acb047c7071bf6c70c6a7ffd4002df446b0"} Nov 02 21:47:48.549720 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:48.549542 5136 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" event={"ID":"eee1d135-b802-45cb-a2cc-6cdcab121f18","Type":"ContainerStarted","Data":"9442bbbd9451c3562b8eab21f0c42b3392670cd8ce3dc50df97ec3ae63bbbfee"} Nov 02 21:47:48.550304 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:48.550286 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:48.659527 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:48.659430 5136 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" podStartSLOduration=1.6594164949999999 podStartE2EDuration="1.659416495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:48.657949445 +0000 UTC m=+7152.264493943" watchObservedRunningTime="2026-11-02 21:47:48.659416495 +0000 UTC m=+7152.265960993" Nov 02 21:47:49.565559 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:49.565525 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-5f2xx_c157ba0d-6496-4ec6-bfad-aafbaf9439f3/manager/0.log" Nov 02 21:47:49.695649 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:49.695613 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w9xl7_73e480fc-41e8-405f-a466-2e08f9041414/server/0.log" Nov 02 21:47:51.013896 ip-10-0-128-34 kubenswrapper[5136]: E1102 21:47:51.013833 5136 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hspfn" containerName="extractor" Nov 02 21:47:52.083657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:52.083622 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2slch_186a3e83-bf4f-4cca-972d-2761964a7c63/manager/0.log" Nov 02 21:47:52.265077 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:52.265048 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-jb6hq_78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b/proxy/0.log" Nov 02 21:47:55.239665 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:55.239628 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656cc65bfd-xlcwc_78556d8e-59a8-4977-8d09-8586ee3cb6c7/manager/0.log" Nov 02 21:47:55.535887 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:55.535810 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4rc8h_45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/kube-state-metrics/0.log" Nov 02 21:47:55.576599 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:55.576571 5136 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-ztpnh" Nov 02 21:47:55.741438 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:55.741408 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mg6sc_95efd56e-e640-4249-b8f4-5e55d11f91c2/manager/0.log" Nov 02 21:47:55.937621 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:55.937542 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4rc8h_45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/kube-rbac-proxy-main/0.log" Nov 02 21:47:56.351443 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:56.351412 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4rc8h_45fb1ec8-149f-4d4b-a5b6-728048a4a0e2/kube-rbac-proxy-self/0.log" Nov 02 21:47:56.767924 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:56.767875 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-995647d55-2ckjz_fd59f213-a474-4fb6-a994-c3dd365fe467/metrics-server/0.log" Nov 02 21:47:56.988645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:56.988607 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94dcf6f94-5f2xx_c157ba0d-6496-4ec6-bfad-aafbaf9439f3/manager/0.log" Nov 02 21:47:57.101483 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:57.101410 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w9xl7_73e480fc-41e8-405f-a466-2e08f9041414/server/0.log" Nov 02 21:47:57.194013 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:57.193980 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-q2q9h_3d6f4284-710c-441c-973d-d61c90b9443f/monitoring-plugin/0.log" Nov 02 21:47:59.601485 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:59.601444 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2slch_186a3e83-bf4f-4cca-972d-2761964a7c63/manager/0.log" Nov 02 21:47:59.765345 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:47:59.765298 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-jb6hq_78e9d3a6-8d7f-49d4-ab1f-9a408c4ae02b/proxy/0.log" Nov 02 21:48:00.099445 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:00.097780 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xb77n_6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/node-exporter/0.log" Nov 02 21:48:00.505645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:00.505615 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xb77n_6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/kube-rbac-proxy/0.log" Nov 02 21:48:00.915145 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:00.915069 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xb77n_6e6c4c6f-add0-4e41-a3aa-c8a48789a1a2/init-textfile/0.log" Nov 02 21:48:01.330374 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:01.330341 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dvlc_0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/kube-rbac-proxy-main/0.log" Nov 02 21:48:01.742019 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:01.741977 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dvlc_0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/kube-rbac-proxy-self/0.log" Nov 02 21:48:02.154171 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:02.154095 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dvlc_0d90c8b2-351c-45f5-a22c-84a64a1a9fd4/openshift-state-metrics/0.log" Nov 02 21:48:08.065065 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:08.065034 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/thanos-query/0.log" Nov 02 21:48:08.475686 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:08.475655 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/kube-rbac-proxy-web/0.log" Nov 02 21:48:08.654406 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:08.654376 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9c5d7d8f-bs77n_46cd203e-7ee4-4aa9-8b62-c5de1807bb2c/kube-auth-proxy/0.log" Nov 02 21:48:08.886645 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:08.886567 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/kube-rbac-proxy/0.log" Nov 02 21:48:09.081809 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:09.080844 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-24nqh_fa795690-3689-45c2-b455-a1b7b58e873d/istio-proxy/0.log" Nov 02 21:48:09.335125 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:09.335095 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/prom-label-proxy/0.log" Nov 02 21:48:09.745108 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:09.745080 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/kube-rbac-proxy-rules/0.log" Nov 02 21:48:10.155657 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:10.155581 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d6d89c5c-4ccnw_a9e00b19-2153-4977-8126-4a5268f7252a/kube-rbac-proxy-metrics/0.log" Nov 02 21:48:10.338544 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:10.338515 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-ns7hq_f17725dd-2440-4da1-93ec-41e643aaf992/istio-proxy/0.log" Nov 02 21:48:10.547332 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:10.547300 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-nzffk_3d7346bd-7650-4820-a6e3-65ef8979ff89/istio-proxy/0.log" Nov 02 21:48:10.969138 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:10.969113 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-z44pn_14f88a84-e518-4c2c-abbd-8bc07a283a4d/istio-proxy/0.log" Nov 02 21:48:11.381460 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:11.381385 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668545c68-6zrh6_04992037-54d2-409a-9364-9d1acab24b15/payload-pre-processing/0.log" Nov 02 21:48:11.590069 ip-10-0-128-34 kubenswrapper[5136]: I1102 21:48:11.590029 5136 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7f8f4d5bdc-f56nj_6c777b53-481c-40bf-a1c2-d2c188988730/payload-processing/0.log"